System and method for movie segment bookmarking and sharing

ABSTRACT

Several ways are provided for a viewer of a movie to create a deep tag, that is, a bookmark for a segment of the movie. The deep tag can be associated with descriptive text and sent to an address provided by the viewer, either an e-mail address or an instant messaging address. Additionally, before the deep tag is created, it can be checked whether the content owner of the movie is known. If known, the content owner&#39;s rules, if any, regarding deep tagging are followed. If unknown, a set of registered content owners can be alerted of the presence of new content. When ownership of the new content is established, the already-created deep tags can be updated in accordance with the content owner&#39;s rules, if any, regarding deep tagging.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. patent application Ser. No. 16/744,840,filed Jan. 16, 2020, now U.S. Pat. No. 10,958,876, entitled “SYSTEM ANDMETHOD FOR MOVIE SEGMENT BOOKMARKING AND SHARING,” which is acontinuation of U.S. patent application Ser. No. 15/365,308, filed Nov.30, 2016, now U.S. Pat. No. 10,594,981, entitled “SYSTEM AND METHOD FORMOVIE SEGMENT BOOKMARKING AND SHARING,” which is a continuation of U.S.patent application Ser. No. 11/713,115, filed Feb. 27, 2007, now U.S.Pat. No. 9,648,281, entitled “SYSTEM AND METHOD FOR MOVIE SEGMENTBOOKMARKING AND SHARING,” which is a continuation-in-part of U.S. patentapplication Ser. No. 11/592,901, filed Nov. 3, 2006, now U.S. Pat. No.8,145,528, entitled “MOVIE ADVERTISING PLACEMENT OPTIMIZATION BASED ONBEHAVIOR AND CONTENT ANALYSIS,” which is a continuation-in-part of U.S.patent application Ser. No. 11/439,600, filed May 23, 2006, now U.S.Pat. No. 8,755,673, entitled “METHOD, SYSTEM AND COMPUTER PROGRAMPRODUCT FOR EDITING MOVIES IN DISTRIBUTED SCALABLE MEDIA ENVIRONMENT,”of U.S. patent application Ser. No. 11/439,594, filed May 23, 2006, nowU.S. Pat. No. 8,724,969, entitled “METHOD, SYSTEM AND COMPUTER PROGRAMPRODUCT FOR EDITING MOVIES IN DISTRIBUTED SCALABLE MEDIA ENVIRONMENT,”and of U.S. patent application Ser. No. 11/439,593, filed May 23, 2006,now U.S. Pat. No. 7,877,689, entitled “DISTRIBUTED SCALABLE MEDIAENVIRONMENT FOR MOVIE ADVERTISING PLACEMENT IN USER-CREATED MOVIES,”which claim a benefit of priority from U.S. Provisional Application No.60/683,662, filed May 23, 2005, entitled “DISTRIBUTED SCALABLE MEDIAENVIRONMENT.” All applications referenced in this paragraph are fullyincorporated herein in their entireties.

BACKGROUND OF THE RELATED ART

The invention relates to processing recorded video and audio, and morespecifically, to techniques for determining where in a movie to insertan ad.

Linear media editing systems used for analog audio, video tape andphotographic film are manual, time consuming and cumbersome to reducecontent into a final form and distribute. In more recent times computersystems allow for time efficient non-linear video editing systems.Current non-linear editing on computer oriented systems involvescapturing media content permitting rapid access to the media content atany point in the linear sequence for moving portions into any order andstoring the media content on a storage device, such as a magnetic diskdrive or digital versatile disk (DVD).

The average person currently has small set of alternatives for editingcontent from media capture devices such as camcorders, camera phones,audio recorders, and other media capture devices without having to incurthe costs of a computer system and software for editing. In addition,non-linear editing systems are complex and very difficult to use.

People capture various random and personally interesting events, such aswork, travel and entertainment event using their camcorders or cameraphones. To edit this content, people require easy to use non-linearediting systems that facilitate editing without a high degree ofcomputer or editing skill.

Media content storage technologies provide for storing great amounts ofinteractive multimedia, for example, the DVD format. Unfortunately, theDVD Specification for authoring is very complex, as are the computersystems that attempt to embody it. A further disadvantage ofconventional DVD authoring systems is that they provide a DVD authorwith only minimal control and flexibility.

The process of authoring a DVD includes a number of complex steps andequipment. Accordingly, there is a need for authoring systems andmethods that reduces the time, cost and complexity of the authoring anddistributing DVD.

Separate and distinct systems for computer based non-linear editing, DVDauthoring, and distributions are known. However, no system exists thatenables an ad to be flexibly placed in different areas of a movieaccording to an automatic procedure.

SUMMARY OF THE INVENTION

In accordance with the present invention, there is provided a method forenabling a viewer to create a deep tag for a segment of a movie. Anindicator for generating the deep tag representing the segment of themovie is provided. An actuation of the indicator is received from theviewer. A deep tag corresponding to the actuation of the indicator iscreated and stored.

In some cases, descriptive text provided by the viewer is associatedwith the deep tag, and the deep tag is sent to at least one addressprovided by the viewer.

It is not intended that the invention be summarized here in itsentirety. Rather, further features, aspects and advantages of theinvention are set forth in or are apparent from the followingdescription and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram for one embodiment of the presentinvention wherein the transcoder is used via a public communicationnetwork;

FIGS. 2A and 2B are configuration diagrams for other embodiments of thepresent invention wherein the transcoder resides locally;

FIG. 3 is an event chart showing distribution and collection of work fora PC-based transcoder using a peer-to-peer network;

FIGS. 4A-4C are configuration diagrams for embodiments of the presentinvention wherein the transcoder resides locally;

FIGS. 5A-5D are charts illustrating various forms of static and proxy adplacement in a movie;

FIG. 6 is an event chart illustrating proxy ad delivery;

FIGS. 7 and 8 are flowcharts depicting high-level operation of movieuploading and downloading, respectively;

FIG. 9 is a flowchart depicting a high level ad placement procedure;

FIG. 10 is a flowchart depicting transcoder operation during moviecreation;

FIG. 11 is a diagram of a user interface for editing a movie;

FIG. 12 is a diagram showing an effects and scripts editor interface;

FIG. 13 is a flowchart illustrating collection and processing of usagestatistics;

FIG. 14 is a timeline illustrating sample play events;

FIG. 15 is a data structure representing the sample play events of FIG.14;

FIG. 16 is a diagram illustrating automatic determination of popularparts of a movie;

FIG. 17 is a flowchart depicting a portion of transcoder operationduring movie creation;

FIGS. 18A-18C are charts illustrating ad bounceback;

FIGS. 19A-19C are charts illustrating slip-ad;

FIG. 20 is a chart illustrating ad bounceback and slip-ad;

FIGS. 21-23 are flowcharts illustrating different techniques for usingpopularity statistics to insert ads in a movie;

FIG. 24 is a diagram illustrating an ad editor interface;

FIG. 25 is a diagram illustrating access permission for a movie;

FIG. 26 is a flowchart depicting transcoder operation to generate aplayback product;

FIGS. 27-32 are diagrams used in explaining specialized editingcapabilities for making playback products;

FIG. 33 is a diagram illustrating a transcoded movie;

FIG. 34 is a data schema used in the present invention;

FIG. 35 is a diagram of a user interface for playing a movie;

FIG. 36 is a configuration diagram for another embodiment of the presentinvention;

FIG. 37 is a flowchart showing creation of a deep tag and sending thedeep tag to another party;

FIGS. 38A and 38B show embodiments of deep tags;

FIG. 39 depicts the process of converting a movie into a texture strip;

FIGS. 40A-40D depict different ways to create a movie-id-print;

FIG. 41 is a diagram of another user interface for playing a movie;

FIG. 42 is a diagram of a pop-up window for creating a deep tag;

FIG. 43 is a diagram of a recipient's screen showing reception of aninstant deep tag; and

FIG. 44 is a flowchart showing details of creating and modifying deeptags.

DETAILED DESCRIPTION

As used herein and in the claims, the term “movie” refers to videoand/or audio data intended for display to a human. In some cases, thevideo and audio data are separate but associated data, while in othercases, the video and audio are combined. In still other cases, videoexists without audio, and vice-versa. Video encompasses still image dataand moving image data.

As used herein and in the claims, the term “editing” includes but is notlimited to removing portions of a movie, combining at least two moviesto create a new movie, and/or generation of special effects that arecomposited into a movie. The term “compositing” refers to combining byinserting or overlaying additional content into movie content, forexample, adding foreign language subtitles.

As used herein, the term “editor” sometimes means a human who is editinga movie, and sometimes means a computer software program used by thehuman who is editing a movie to perform the editing.

The disclosures of the following patents are hereby incorporated byreference in their entirety:

U.S. Pat. No. Issue Date Title 5,880,722 Mar. 9, 1999 Video cursor withzoom in the user interface of a video editor 5,886,692 Mar. 9, 1999Attracting/repelling edit blocks in a user interface of a video editor6,027,257 Feb. 22, 2000 Pan and tilt unit 6,157,771 Dec. 05, 2000 Methodand apparatus for seeking within audiovisual files 6,181,883 Jan. 30,2001 Dual purpose camera for VSC with conventional film and digitalimage capture modules 6,201,925 Mar. 13, 2001 Method and apparatus forediting video files 6,262,777 Jul. 17, 2001 Method and apparatus forsynchronizing edited audiovisual files 6,285,361 Sep. 04, 2001 Methodand apparatus for clipping video segments from an audiovisual file6,400,886 Jun. 04, 2002 Method and apparatus for stitching edited videosegments 6,661,430 Dec. 09, 2003 Method and apparatus for copying anaudiovisual segment

There are several environments in which a user might want to use amovie. Turning to FIG. 1, an environment including public communicationnetwork 100 is shown. Network 100 may be the Internet. Coupled tonetwork 100 are movie system 10, server 80, distributed server 82, proxyad server 84, media center 105, PC 110, phone 130, PC 140 and camcorder145. Coupling occurs via wireline and/or wireless communication lines,possibly with intermediate computers such as an Internet serviceprovider (not shown).

Movie system 10 includes videobase 20, database 30, transcoder 40,server 50, DVD burner 60 and internal network 70. In some embodiments,videobase 20 and database 30 are combined. Elements 20, 30, 40, 50 aregeneral purpose computers programmed according to the present invention,and include suitable processors, memory, storage and communicationinterfaces; and each element may be embodied in one or many physicalunits depending on factors such as expected processing volume, redundanthardware design and so on.

Videobase 20 serves to store movies uploaded by users, in their uploadedformat, and to store transcoded versions of these movies. Videobase 20also stores advertisement movies, referred to herein as “ads”, intendedfor inclusion in the transcoded movies.

Database 30 serves to store data for movie system 10, including datarelating to users of movie system 10, movies processed by movie system10, and suitable administrative data such as usage, throughput and audittrail information. In some embodiments, users use movie system 10 forfree and the suppliers of ads pay upon ad viewing. In other embodiments,users pay based on usage or a flat rate.

Transcoder 40 serves to receive uploaded movies and process them togenerate transcoded movies, as described in detail below with regard toFIG. 10.

Server 50 receives requests from users via network 100 and respondsthereto. In cases where responding to a request requires the services oftranscoder 40, server 50 passes appropriate messages between transcoder40 and network 100. Server 50 also functions as a firewall to protectnetwork 70 from improper usage.

Server 50 executes upload manager 55, a software program that works withuploader 112, described below, to upload a movie to server 50.

DVD burner 60 is responsive to commands from transcoder 40 to create adigital video disk, which is then shipped via conventional physicalshipping services. In other embodiments, instead of a digital videodisk, a read-only or read-write storage medium is created and then themedium is physically delivered to a recipient. Suitable formats includeCD (compact digital), DVD (digital video disk), Blu-ray Disc, HD (highdefinition) DVD, flash drives (transistor memory), jump drives(removable disk drives) and the like.

Billing program 65 examines usage data created by transcoder 40 andserver 50. The usage data is part of the administrative data in database30. Billing program 65 then generates invoices and applies authorizedpayments. For example, some users may have preauthorized charges totheir credit cards, telephone bills or bank accounts for their usage ofmovie system 10. As another example, transcoded movies created by usersmay include advertising, for which the advertisers agree to pay based onnumber of views, and if so, billing system 65 arranges payments to usersbased on usage of the transcoded movies with advertising.

There is a cost to store and distribute movies. To offset this cost, andto reward users, movie system 10 enables movie creators to include ads,either manually or automatically, in their movies. Movie system 10enables flexible ad placement, including at the start or end of a movie,within selected frames of a movie, and at a selected location and sizewithin the selected frames. Advertisers generally pay for placement oftheir ads based on number of times their ad is viewed, and possibly inaccordance with the popularity of the place in the movie where the ad isinserted, how much time the ad is inserted for, and the size of the adrelative to a movie frame.

Internal network 70 serves to carry communication traffic between theelements of movie system 10. Internal network 70 may be a local areanetwork at a single premises, or may span multiple premises.

Server 80 is a general purpose computer coupled to storage 90. Server 80responds to requests from communication network 100 by providing moviesstored in storage 90. By providing the address of server 80 to moviesystem 10, one of the movies stored in storage 90 can be used as aninput for transcoder 40.

Distributed server 82 is a general purpose computer coupled to storage92. Storage 92 holds copies of some or all of the transcoded moviesstored in storage 20 of movie system 10. In some embodiments, there aremany instances of server 82 and storage 92. Distributed server 82responds to requests from viewers for the transcoded movies storedtherein. Use of distributed server 82 reduces response time to requests,reduces the amount of work that movie system 10 needs to do, and reducesbandwidth utilization since usually the instance of server 82 that isclosest to the requestor responds thereto. Products and servicesaccording to distributed server 82 are available from Kontiki, aVerisign company, and from Akamai.

Proxy ad server 84 is a general purpose computer coupled to storage 94.Storage 94 stores ads. Operation of proxy ad server 84 is discussedbelow with regard to FIG. 6.

Media center 105 generally represents a television that is able tocommunicate via network 100. In some embodiments, a viewer providesinstructions to media center 105 via a remote keypad, while in otherembodiments, a viewer provides instructions to media center 105 using afull keyboard with function keys and so on. In still other embodiments,media center 105 includes a voice recognition unit so that the viewerprovides instructions through speech. In some embodiments, such as ahome, media center 105 is able to communicate directly with PC 110and/or phone 130.

PC 110 is a general purpose personal computer coupled to camcorder 120.Camcorder 120 enables a user to record a movie and transfer the movie toPC 110.

PC 110 executes uploader 112 and player 114. Uploader 112 is a softwareprogram that enables a movie to be uploaded from PC 110 to server 50.Player 114 is a software program that enables PC 110 to view and editmovies, in conjunction with transcoder 40. When PC 110 registers withserver 50, server 50 downloads uploader 112 and player 114 to PC 110.

Uploader 112 functions to locate movie files stored in PC 110, and tomanage transmission of the movie files to upload manager 55 of server 50using a suitable protocol such as the secure file transfer protocol(sftp). In embodiments having a peer-to-peer network for downloading,such as networks using the bittorrent protocol, the peer-to-peer networkis also used for uploading. Since movie files are large, the fileuploading may be interrupted; uploader 112 enables the uploading toresume at its interruption point. In some embodiments, uploader 112converts a very large file, such as a 36 Mb file in DV format, to asmaller file of comparable visual quality, such as a 3 Mb files in MPEGformat. Uploader 112 enables the user of PC 110 to select a file foruploading; to monitor the status of the upload, such as percentcompleted and speed of uploading; to pause and resume the uploading; andto cancel the uploading.

Player 114 is a client application for a PC browser. In someembodiments, player 114 resides on a portable device such as a mobilephone or network-attached digital video camera without a browser, and inthese embodiments, player 114 is a network enabled client application.

Player 114 enables a user to view a movie, including forward seek andrewind functions; to seek the compressed video on a remote server usingrandom seek points; to request a movie download from the random seekpoint, for example, in accordance with U.S. Pat. No. 6,157,771, thedisclosure of which is hereby incorporated by reference in its entirety;and to use the functions described below with regard to FIG. 10 on themovie, including creating and editing deep tags (movie segmentbookmarks, see FIG. 38A), creating and editing mash-ups, adding specialeffects, providing sharing permission to other users, creating virtualand/or physical DVDs, inserting ads, and creating and editingwatermarks. FIG. 11, discussed below, shows a graphical user interfacefor player 114.

Phone 130 is a wireless communication device executing versions ofuploader 112 and player 114 adapted for the device capabilities of phone130. Phone 130 is a coupled to camera 135, which serves to captureimages and provide the captured images to phone 130 as a movie signal.In some embodiments, phone 130 uses the multimedia messaging service(MMS) protocol to transmit and/or receive movies. In other embodiments(not shown), phone 130 communicates with a local network using aprotocol such as WiFi, and the WiFi network in turn communicates withcommunication network 100.

PC 140 is a general purpose personal computer that is able to viewtranscoded movies by obtaining an address for the movie, or a segmentthereof, and providing the address to server 50. As an example, a userof PC 110 or a user of phone 130 may upload a movie to movie system 10,edit the uploaded movie, and provide to the user of PC 140, via email,an address of an edited segment that the user of PC 140 is permitted toview.

Camcorder 145 is a network enabled movie capture device configured toupload its recordings to movie system 10. In some embodiments, there isat least one predefined user group able to immediately edit informationuploaded from camcorder 145. This configuration is useful in a securityapplication, for example.

The user of PC 110 or phone 130 serves as an editor. PC 110, phone 130,PC 140 and camcorder 145 are each at respective locations that areremote from the location of movie system 10.

The capabilities of movie system 10 are available via each of the userdevices shown in FIG. 1.

Other embodiments of the present invention wherein the transcoder islocal to the movie file are described below. Embodiments in which themovie file is not uploaded via a public communication network aresometimes referred to as a transcoder appliance model.

FIG. 2A shows PC 141 as executing transcoder 142. PC 141 typicallyreceives at least one movie to be edited, such as from server 80, anduses transcoder 142 to edit the movie. In some embodiments, transcoder142 provides only a subset of the functions provided by transcoder 40,in accordance with the processing capability of PC 141. After creating atranscoded movie, PC 141 uploads the transcoded movie to storage 20 inmovie system 10, such as by connecting to communication network 100.

FIG. 2B shows a peer-to-peer transcoder network. PCs 143A, 143B, 143Care connected to communication network 100 and respectively executetranscoders 144A, 1434B, 144C. Since movie editing can involvesubstantial processing, each of transcoders 144A, 144B, 144C is adaptedto distribute parts (tasks) of an editing job to others of transcoders144A, 144B, 144C. Of course, each of transcoders 144A, 144B, 144C isadapted to receive an editing task, to execute the task, typically aspart of background processing, and to send the completed task back tothe originating transcoder.

FIG. 3 is a flowchart illustrating peer-to-peer transcoding. Let it beassumed that the user of PC 143B is editing a movie. At some point, theuser will command a job that transcoder 144B recognizes as requiringsubstantial processing. Examples are applying a special effect to eachframe of a movie, applying a special effect to combine one movie withanother such as picture-in-picture (PIP), and compositing of adplacement objects such as cars, clothing, beverages etc. into a moviescene, and so on.

At step 200, transcoder 144B divides the job into tasks. At step 202,transcoder 144B sends one task to transcoder 144A. At step 206,transcoder 144B sends another task to transcoder 144C. At step 210,transcoder 144B receives the completed task from transcoder 144A. Atstep 211, transcoder 144B receives the completed task from transcoder144C. At step 212, transcoder 144B assembles the completed tasks into acompleted job.

At step 203, transcoder 144A receives a task. At step 204, transcoder144A executes its task. At step 205, transcoder 144A sends the executedtask back to transcoder 144B.

At step 207, transcoder 144C receives a task. At step 208, transcoder144C executes its task. At step 209, transcoder 144C sends the executedtask back to transcoder 144B.

Turning to the transcoder appliance model, FIG. 4A shows PC 150,camcorder 160 and DVD burner 170. Camcorder 160 is for recording a movieand providing the recorded movie to PC 150. PC 150 is a general purposepersonal computer operative to receive and store a recorded movie; toenable a user to edit the stored movie using transcoder 155, which is asoftware program operative in a similar manner as transcoder 40; and toprovide a transcoded movie to DVD burner 170 for recording thereon, insimilar manner as DVD burner 60.

FIG. 4B shows video recorder 180, robotic camera with pan-tilt-zoom(PTZ) control 182, and DVD burner 183. Robotic camera 182 is operativeto provide a video signal to video recorder 180. Video recorder 180 is ageneral purpose computer operative to receive and store the video signalfrom robotic camera 182, to enable a user to edit the stored video usingtranscoder 181, which is a software program operative in a similarmanner as transcoder 40; and to provide a transcoded movie to DVD burner183 for recording thereon, in similar manner as DVD burner 60.

FIG. 4C shows robotic camera 191, video recorder 190, private network192 and PC 193. PTZ camera 191 is operative to provide a video signal tovideo recorder 190. Video recorder 190 is operative to receive and storethe video signal from robotic camera 191, and to provide the storedvideo to PC 193 via private network 192. Private network 192 is a localarea network, virtual private network, or the like, which serves tocouple video recorder 190 and PC 193. PC 193 is a general-purposepersonal computer operative to receive video from video recorder 190,sometimes in response to a command from PC 193 and sometimes on aperiodic or as-available basis from video recorder 190; to store thereceived video; and to execute transcoder 194 to edit the stored videoto produce transcoded video. Transcoder 194 is a software programoperative in a similar manner as transcoder 40.

FIG. 5A is a chart showing static ad 93A in a mid-roll position in movie20A. Movie 20A comprises frames of video presenting content, such as acomedy movie. During playback, at a particular frame, the comedy moviestops, and the frames of ad 93A are played back, then the frames of thecomedy movie resume playback. Conventionally, ad 93A is placed at thestart of the comedy movie, at the end, or at an intermediate point, asshown in FIG. 5A.

FIG. 5B shows static ad 93B, which is combined with frames of movie 20Bto produce composite video. Static ad 93B may be placed, e.g., as abanner at the top, and may show a graphic image or an advertising movie,and so on.

FIG. 5C shows proxy ad 94A at a mid-point position of movie 20C. Duringplayback, at the start of the movie, a request is sent to proxy adserver 84, which replies by sending ad 94B. At a particular frame, thecomedy movie stops, and the frames of ad 94B are displayed, then theframes of the comedy movie resume. See FIG. 6 below.

FIG. 5D shows original proxy ad 94C superimposed on frames of movie 20D.Original proxy ad 94C may be placed, e.g., as a banner at the top.During playback of movie 20D, ad 94D is requested from proxy ad server84, and the frames of ad 94D are combined with the frames of movie 20Dto produce composite video that is displayed.

FIG. 6 is an event chart illustrating proxy ad delivery. Let it beassumed that the user of PC 140 wants to view a transcoded movie storedin movie system 10, and that the transcoded movie includes therein aproxy ad, with the ad itself residing in storage 94.

At step 220, PC 140 requests the movie from movie system 10. At step221, movie system 10 receives the request, and at step 222 respondsthereto by sending the movie including the proxy ad to PC 140. The proxyad includes an ad request that PC 140 is expected to send. The adrequest typically comprises the network address of proxy ad server 84,and optional or required parameters to be used by proxy ad server 84such as movie characteristics and/or viewer characteristics, e.g.,keywords relating to the movie, display characteristics of PC 140,vertical and horizontal picture resolution of an ad image that can beaccommodated in the movie, and so on.

At step 223, PC 140 receives the movie and ad request. At step 224, PC140 sends the ad request and parameters, if any, to proxy ad server 84.At step 225, proxy ad server 84 receives the ad request and parameters,if any. At step 226, proxy ad server 84 selects an ad from storage 94 inaccordance with the parameters, and prepares the ad, such as by sizing,in accordance with the parameters. At step 227, proxy ad server 84 sendsthe prepared ad to PC 140.

At step 228, PC 140 receives the prepared ad, and plays the movie whileinserting the received ad at the location indicated by the proxy ad inthe movie.

FIGS. 7-8 are flowcharts depicting high-level operation of movieuploading and downloading, respectively.

Turning to FIG. 7, at step 300, user1 (not shown) at PC 110 uploads amovie to server 50, such as a movie recorded by camcorder 120. Theuploaded movie is in an original format, also referred to as a firstformat or an uploaded format. In some cases, user1 provides an addressof server 80, and instructs server 50 to retrieve the movie from server80. Server 50 stores the uploaded movie in videobase 20. At step 310,transcoder 40 processes the uploaded movie, enabling the user to editthe uploaded movie as described in detail below, to produce a transcodedmovie. The transcoded movie is in a proxy format, also referred to as asecond format or an intermediate format.

At step 320, transcoder 40 stores the transcoded movie in videobase 20.As described below, user1 can grant permission to other users to viewthe whole of the transcoded movie, or to view segments of the transcodedmovie, by providing suitable addresses to the authorized users.

Turning to FIG. 8, at step 330, user2 (not shown), who has receivedpermission to view the transcoded movie, sends a request for the movieto server 50. The request includes information about the displaycapability of the device used by user2, such as phone 130 or PC 140.

At step 340, server 50 requests the selected movie from videobase 20,indicating the format for the movie.

At step 350, if the requested format happens to match the stored format,then the movie is provided directly to server 50. Otherwise, videobase20 is operative to convert the format from the stored format to therequested format. The movie is provided in the requested format, alsoreferred to as a third format, a downloaded format, or an output format.Server 50 then sends the provided movie to user2 via suitabledistribution method such as streamed video or podcast, or presentationon a web page, blog, wiki, really simple syndication (RSS) or othertechnique. In some embodiments, videobase 20 sends the stored movie totranscoder 40, for conversion to the requested format, and thentranscoder 40 provides the movie in the requested format to server 50.

FIG. 9 depicts the overall ad placement process.

At step 2005, transcoder 40 analyzes the movie characteristics. As usedherein and in the claims, inherent movie characteristics meansinformation about the movie itself, without reference to usage orviewing. Examples of inherent movie characteristics include motion,scene changes, face presence, audio track loudness, keywords relating tothe movie's subject and so on. In this document, inherent moviecharacteristics are sometimes referred to as movie characteristics.

At step 2010, transcoder 40 determines whether the movie has beenpreviously viewed.

If not, processing proceeds to step 2020. If the movie has beenpreviously viewed, then at step 2015, transcoder 40 analyzes the viewedcharacteristics of the movie, also referred to herein as the popularityof the movie. Viewed characteristics include number of times that themovie has been requested, number of times that a link or deep tag to themovie has been sent from one viewer to another potential viewer, numberof deep tags in the movie, number of times that a particular moviesegment was replayed, number of times that viewers paused at aparticular movie position, and so on. In some embodiments, the viewedcharacteristics are associated with the demographics of the viewer, suchas gender, age, location, income, interests and so on.

At step 2020, transcoder 40 determines if the ad placement is occurringin real time. As explained below, movie system 10 can operate in one ormore of the following modes:

-   -   by placing one or more ads in a movie, storing the movie plus        ads, and then delivering the stored movie plus ads to a viewer        in response to the viewer request, referred to as non-real time        ad placement mode;        -   As explained below, non-real time ad placement mode has a            static version and a variable version. In the static            version, the ad is placed once, and remains in its place for            the duration of the advertising period, i.e., the timeframe            that the advertiser pays for. In the variable version, the            ad's position may change from time to time, in accordance            with changes in the viewed characteristics of the movie and            changes in other advertiser's interest in having their ads            included in the movie.    -   by receiving a viewer request for a movie, then placing ads in        the movie and delivering the movie plus ads to the viewer,        referred to as real-time ad placement mode;    -   by placing a first set of ads in the movie, storing the movie        plus ads, receiving a viewer request, then placing a second set        of ads in the movie, and delivering the movie with the first and        second sets of ads to the viewer, referred to as real time plus        non-real time ad placement mode, or hybrid ad placement mode.

Non-real time ad placement mode enables movie system 10 to operate inessentially a broadcast fashion, for ads. Real-time ad placement modeenables movie system 10 to operate in a narrowcast fashion, wherein thecharacteristics of the viewer determine what ads are sent to the viewer.If movie system 10 is operating in non-real time ad placement mode, thenprocessing proceeds to step 2030. If movie system 10 is operating inreal-time ad placement mode, or hybrid ad placement mode, then at step2025, transcoder 40 analyzes the viewer's characteristics. In thiscontext, characteristics of the viewer includes demographic informationas well as activity of the viewer, such as entering a keyword, thematerial previously viewed by the viewer and so on.

At step 2030, transcoder 40 retrieves the advertiser's preferences forad placement. Advertiser preferences can specify movie characteristics,movie popularity characteristics, viewer characteristics, and how muchthe advertiser is willing to pay depending on the combination offeatures delivered, which may include whether the ad is viewed orwhether the viewer takes an action relating to the ad. For example, someads include hyperlinks that a viewer can click on, and the advertisermay be willing to pay a first rate if the ad is merely presented to theviewer, and a second rate if the viewer clicks on the hyperlink.

At step 2035, transcoder 40 determines prices for positioning ads atvarious places in the movie. The price determination procedure is afunction of the movie characteristics, the viewed characteristics, theviewer characteristics, advertiser demand, and the price settingmechanism.

At step 2040, transcoder 40 places the at least one ad in the movie. Itwill be appreciated that an advertiser's rules may specify manualplacement, in which case transcoder 40 notifies a person associated withthe advertiser of the prices for placing the ad at various positions,and a human manually selects ad placement.

In some cases, an ad is composed into the scene, usually manually. Forexample, the movie scene might be a living room with a television, andthe ad is composed to appear to be on the television in the movie scene.As another example, the movie scene may be a wall with a picture frame,and the ad is composed to appear in the picture frame of the moviescene.

At step 2045, the movie is delivered to the viewer that requested themovie. When movie system 10 operates in non-real time ad placement mode,there is a step (not shown) of storing the movie plus ads in videobase30 prior to delivering the movie to the requesting viewer.

At step 2050, statistics are collected about movie viewing, such as byserver 50 receiving play trails (discussed below), notices of deep tagtransmissions and so on, and storing them in database 30.

At step 2055, appropriate viewing reports are generated by billingprogram 65, server 50 and transcoder 40.

A prior art search engine enables advertisers to bid on keywords. Duringa set-up phase, the advertiser selects the desired keyword(s), and theirmaximum bid price. In operation, when a searcher enters the desiredkeyword, advertiser ads are presented in an order corresponding to theirbid amounts, that is, the ad with the highest bid is listed first,followed by ads with sequentially lower bid amounts. When the searcherclicks on an ad, the search engine bills the advertiser, that is, if thead is merely presented with no response from the searcher, theadvertiser does not pay. Advertisers can then view reports on the“click-through” activities for their advertisements.

In an embodiment wherein the price of placing an ad in a movie varies,advertisers can determine the price at least in part by bidding. Forexample, movie system 10 can determine a starting price based for adplacement in portions of a movie based on popularity, level of motion,length of ad, whether the ad is full-motion video or a static image, andso on. Advertisers can then (i) submit specific bids for specificmovies, ex: $100 for placement in movie xyz for two weeks, (ii) candefine bidding rules, ex: placement in any travel movies viewed at least100 times per day, ex: placement in any movie segment having a deep tagwith the text label “airplane” and where the movie is one of the mostpopular 1,000 movies for the day, or (iii) can define results and howmuch they are willing to pay, ex: at least 100 views per hour by girlsof age 10-14 located in California at a cost of up to $20 per hour.

The value placed upon media inventory across a timeline of a movie canvary dramatically and dynamically based on continuously updatedstatistics on both behavioral and contextual analysis of the videostream; for example the most prized video segments may be those that aremost popular and contain the deep tags for “cars”, “hotels”, “swimming”,etc.

FIG. 10 is a flowchart depicting transcoder 40 operation.

As an overview, transcoder 40 receives an uploaded movie, creates arepresentation for easy editing, and adds user-supplied editing data(steps 400-440) to create a transcoded movie. Then, at the user'soption, some, all or none of the following functions can be performed,in any desired sequence, and in as many editing sessions as desired:

-   -   adding special effects (steps 470-490),    -   including advertising movies in the transcoded movie (steps        500-600),    -   providing permission for other users to view all of the        transcoded movie, or segments of the transcoded movie (steps        610-630), and    -   creating a playback product using the transcoded movie, the        playback product being one of a mash-up, a tree or a link-up        (steps 631-632).

Including an advertising movie in a transcoded movie ensures that evenif the viewer of the transcoded movie has a filter for blockingadvertising pop-ups and the like, the included advertising movie isviewed, since the filter considers the advertising movie to be part ofrequested content.

At step 400 of FIG. 10, transcoder 40 receives the uploaded video,either from the user or from another web site. The video may be in oneof several formats, some of which involve compression using a schemewherein a compressed frame refers to an earlier and/or later frame, andthen provides change information relative to the earlier and/or laterframes. Transcoder 40 converts the uploaded movie to a series ofself-referential (uncompressed) frames. Generally, a frame is acollection of picture elements (pixels) corresponding to an imagepresented to a human viewer.

At step 410, transcoder 40 builds a texture strip representing themovie. Specifically, transcoder 40 applies a function to each frame togenerate texture data (see FIG. 39), and saves the texture data as avideo image. For example, the function might be to extract the center8.times.8 pixels of each frame and realign into a 64 pixel height columnand the texture strip is the sequence of 64 pixel columns. The texturestrip may be saved as a .jpg file. The texture strip serves to representthe entire movie in a convenient information bar, and is sometimesreferred to as a navigation bar. The texture strip is an intuitive wayof determining the temporal position of a frame relative to the entiretyof a movie. The texture strip often is useful in detecting scenechanges, which is important when deciding which frames to group togetheras a segment.

At step 420, transcoder 40 creates a source proxy for the uploadedmovie. Generally, a source proxy is a representation of the frames ofthe movie in a particular format that is easy to convert to otherformats and to distribute via public communication network 100. Forexample, the Flash video format, according to the H.263 standard, can beused for the source proxy.

Using a source proxy reduces the format conversion issue. Specifically,if there are n movie formats, a general transcoder should be able toconvert from any input to any output format, which, by brute force,would require n.sup.2 different format converters. However, using asource proxy means that only 2n format converters are needed (nconverters to the source proxy format, and another n converters from thesource proxy format to the output format). Additionally, as new movieformats become available, supporting them requires creating only 2converters per format (one to the source proxy format, and one from thesource proxy format), rather than 2n with the brute force approach. Itis recognized that, sometimes, the source proxy format may be thedesired output format.

Editing of the proxy format, also referred to as proxy editing, mayoccur in several ways.

In one embodiment of proxy editing, the edits are applied directly tothe proxy frames.

In another embodiment of proxy editing, the proxy frames are maintainedas generated, and an edit list is created, comprising edits to besequentially applied to the proxy frames. Each time the edited movie isprovided, the edits are applied anew to the proxy frames. Thisembodiment is particularly useful when edits need to be undone, or whenmany users are editing one movie to create separate edited movies.

In a further embodiment of proxy editing, a hybrid approach is used,wherein during an edit session, an edit list is created, and only at thetermination of the edit session are the edits applied directly to theproxy frames.

At step 430, transcoder 40 generates a thumbnail, also referred toherein as a snapshot, as a visual representation of the entire movie.Typically, the user selects a frame, and transcoder 40 reduces it to athumbnail size, such as 177.times.144 pixels. A user having many storedmovies can conveniently view their thumbnails, rather than or inaddition to text descriptions and/or filename descriptions.

At step 440, transcoder 40 accepts metadata from the user. Moviemetadata may include a filename for the transcoded movie, subject matterkeywords associated with the movie, a short text description to beassociated with the thumbnail, any deep tags the user cares to define,address information such as a hyperlink of information to be associatedwith the transcoded movie, and an associated movie such as an audio filedescribing the contents of the movie.

A deep tag is a video bookmark, indicating a sequential group of framesthat are to be treated as a separately addressable segment; the deep tagmetadata includes the movie filename, the user filename, date ofcreation of the deep tag, date of most recent modification of the deeptag, a deep tag filename, the start frame, the end frame, the durationof the segment, and a short text description of the segment. A deep tagis understood to be a convenient way of identifying a segment. A deeptag can be thought of as a movie segment bookmark.

FIG. 11 shows screen display 700 of player 114, provided, for example,at PC 110, and including video display 710, thumbnail 715, texture strip720, positioner 730, deep tag marker 735, deep tag button 740, deep tagbar 750, and function buttons 760, 771, 772, 773, 775, 780 and 790.Screen display 700 is typically used by an editor.

Video display 710 shows the current frame of video. When the editor'sdevice, such as PC 110 or phone 130 permits, the video frame isdisplayed in its proxy format. However, if the editor's device cannotsupport the proxy format, the transcoder 40 converts edited frames to anoutput format suitable for the editor's device prior to sending theedited frames to the editor for display.

Thumbnail 715 is a small image representing the entire movie.

Texture strip 720 comprises sequential frame representations 725 andsubsequent information; each frame representation 725 is the result ofthe function used to create the texture strip, such as a vertical columnof 64 pixels, and represents a single frame. Subsequent informationindicates special effects applied to the frames and any advertisinginserted in the frames. Texture strip 720 is not limited to videotransform visualizations, and can be created by any of the followingtechniques, or other suitable techniques:

-   -   frequency analysis, such as a spectrogram, wherein spectral        analysis is performed on an input signal, then the amplitude of        the signal at one or more frequencies is plotted against time. A        two-dimensional spectrogram plots the amplitude of one frequency        versus time; a three-dimensional spectrogram has axes of time,        frequency and signal level;    -   other video and/or audio signal transforms, such as wavelets;        and    -   heatmaps (described below).

Positioner 730 indicates where the frame display in video display 710 islocated relative to the entirety of the movie. Positioner 730 enablesthe editor to use texture strip 720 to seek frames in the movie in arandom access manner.

Deep tag marker 735 has a left edge that can be adjusted by a user, andalso has a right edge that can be adjusted by the user; after the userhas adjusted the left and right edges of deep tag marker 735, the userindicates that these settings should be saved as a deep tag, such as byclicking deep tag button 740, and providing a text descriptioncorresponding to the movie segment indicated by the deep tag. Deep tagmarker 735 enables the editor to use texture strip 720 to select asegment of the movie.

Deep tag bar 750 is a visual representation of deep tags that havealready been created for the movie. In the example of FIG. 11, deep tags751 and 752 have previously been created for the movie, and are locatednear the start and end, respectively, of the movie being edited.

Function buttons 760, 771, 772, 773, 775, 780 and 790 enable the user toedit the movie. Data button 760 enables the user to view and editmetadata associated with the movie. Playback product buttons 771, 772and 773 take the user to specialized editing interfaces, discussedbelow. Effects/scripts button 775 enables the user to add and editspecial effects and scripts. Ad button 780 enables the user to includeadvertising in the movie. Sharing button 790 enables the user to grantpermission to other users or user groups to view selected segments ofthe movie.

Returning to FIG. 10, at step 470, transcoder 40 determines whether theuser wishes to apply any special effects and/or add scripts to thetranscoded movie; a script is executable program code associated with amovie or a position in a movie. An example of a scripting capability isActionScript2 for the Macromedia Flash player, that makes it possiblefor Flash movies to be interactive. Other examples of scriptinglanguages are scalable vector graphics (SVG), a language for describinggraphics in extensible markup language (XML), and Javascript. Othersuitable scripting languages will be apparent to those of ordinaryskill. For example, the user indicates a desire to apply special effectsand/or add scripts by selecting effect/script button 775 in FIG. 11. Ifso, at step 480, transcoder 40 provides the user with an effects andscripts editor (discussed below with regard to FIG. 12). In someembodiments, there are about 300 different effects on the effects menu.The user selects desired effects and/or scripts and indicates where inthe movie they should be applied. At step 490, transcoder 40 applies theeffects to the movie and/or associates the scripts with the movie.Generally, the user views the results of the effects, and continueseffects editing until satisfied with the result. The user can createmultiple versions of a transcoded movie, differing in their specialeffects and/or scripts.

Examples of special effects include: watermark, mosaic, barndoor, noise,dissolve, spiral, fade in, fade out, increase contrast, decreasecontrast, soften perimeter, cut frame, overlay, translucency, wipes,cuts, zooms, and so on.

A movie watermark for a movie including video is one of (a) a permanentgraphic such as a logo inserted in each frame of the movie, (b) anoverlay graphic such as a logo inserted in each frame of the movie, or(c) hidden information such as a logo, also referred to assteganography. A movie watermark for an audio only movie is (i) a soundfile inserted at a predetermined point in the movie, such as its startor end, or (ii) hidden information such as text, also referred to assteganography.

Examples of scripts include: when the movie is fully loaded, then play asound file; when the cursor clicks on an area, then create a pop-upwindow with a particular web page; when the cursor clicks on an objectin a video scene, then launch a form—such as for product purchase orsurvey; and so on.

FIG. 12 shows effects and scripts editor interface 1000. Effects area1010 includes library effect button 1020, custom effect button 1021,library script button 1022 and custom script button 1023. Typically, aneffect button has an image of the effect and holding the cursor on thebutton displays text describing the effect, and a script button has anicon on the button and holding the cursor on the button displays textdescribing the script. Some scripts also include effects, and someeffects also include scripts. Forward and reverse navigation buttons1011, 1012 enable navigation among the effects and scripts buttons.Clicking describe button 1013 displays text describing the button (sameas holding the cursor on the button). Clicking show button 1014 createsa pop-up window showing the effect and/or script in action, useful whenthe size of the button is too small to fully convey the effect.Effect/script name window 1015 enables the editor to type the name of aneffect or to select it from a drop-down menu.

An effect can also include an executable command, as described abovewith respect to the tree playback product editor.

Texture strip 1030 indicates the frame representations of the movieselected for effects, such as frame representation 1031. Positioner 1032enables rapid selection of a frame of the movie. Slider 1035 has a leftand right edge that can be adjusted by the editor, to indicate frames toreceive an effect. After the editor selects an effect in effects area1010, and adjust slider 1035, the editor clicks apply button 1033 toapply the effect. After the editor has finished applying effects, he orshe clicks preview button 896 to preview. If an editor wishes to cancelan effect, she positions slider 1035 on the appropriate framerepresentations and clicks cancel button 1034. Publish button 897, burnbutton 898, and back button 899 function as described above.

Returning to FIG. 10, at step 500, transcoder 40 obtains the user'spreference for including advertising in the transcoded movie. Forexample, the user indicates a desire to include advertising by selectingadvertising button 780 in FIG. 11. Videobase 20 contains advertisingmovies, also referred to as ad movies, that is, segments provided byadvertisers for inclusion in user-created movies. Database 30 includesthe advertiser's preference for the types of movies that the advertiserwants its ad movies to be combined with, whether the advertiser prefersor requires that its ad be used as a static insert or a dynamic insert(discussed below), whether the advertiser permits its ad movies to beused as part of a mash-up video, and how the advertiser pays the user,such as a one-time fee, or per-viewing of the ad movie. Conventionalsystems place advertising only at the start of user-created content;movie system 10 is more flexible in that advertising can be insertedwithin a user-created transcoded movie in either a static or a dynamicmanner.

If the user wants to include an ad movie in the transcoded movie, atstep 510, transcoder 40 determines whether the user wishes to select thead movie, or to accept an ad movie selected by movie system 10. If theuser wishes to select the ad movie, at step 520, transcoder 40 providesthe user with a menu of ad movies that are consistent with thecharacteristics of the user's transcoded movie, and the user selects oneor more ad movies for inclusion in their transcoded movie. If the userdoes not wish to select the ad movies, at step 530, transcoder 40selects one or more ad movies based on an ad movie selection procedureand its own determination of how many ad movies are appropriate for thetranscoded movie. In some embodiments, the ad movie selection procedureis based on maximizing revenue for the user, and following aleast-recently-used ad movie selection procedure. In some embodiments,the determination of how many ad movies are appropriate is based on atleast one of: the length of the transcoded movie, keywords in themetadata, how many segments are deep tagged, and the length of the deeptagged segments, and so on.

An ad used with movie system 10 can be static or dynamic. A static ad isinserted in the transcoded movie prior to its storage in videobase 20.For a dynamic ad, the transcoded movie is stored in videobase 20 with aplaceholder, and when the transcoded movie is presented to a user, theactual ad is inserted, the ad being chosen based on a characteristic ofthe user such as the user's location, referred to as a “geo-aware” ad,the characteristics of the display device, referred to as a “deviceaware” ad, or other suitable characteristic.

At step 540, transcoder 40 determines whether the user wishes to controlthe position of the ad movie(s) within a frame of the transcoded movie,or to accept positioning determined by movie system 10. If the userwishes to control the positioning of the ad movie, at step 550,transcoder 40 provides the user with a graphical interface forcontrolling ad movie positioning, and a menu of how the ad-movie shouldbe inserted, such as a picture-in-picture at the top, left, right orbottom of the transcoded movie, the top being referred to as a “banner”ad, or as a stand-alone segment in the transcoded movie. In someembodiments, transcoder 40 also provides the user with popularitystatistics for portions of the movie, discussed in detail below, so thatthe user can position the ad in accordance with the parts of the moviethat viewers like. In some embodiments, the user selects the size of thespace in each frame that the ad may occupy. If the user does not wish tocontrol the positioning of the ad movie, at step 560, transcoder 40decides where the movie ad should be placed, typically by looking at thedefault position in the metadata associated with the ad movie.

At step 570, transcoder 40 determines whether the user wishes to controlthe frames of the transcoded movie where the ad will be placed, or toaccept positioning determined by movie system 10. If the user wishes tocontrol the positioning of the ad movie, at step 580, transcoder 40provides the user with a graphical interface for controlling ad moviepositioning, such as a texture strip and slider, and in some cases,popularity statistics (discussed below) for portions of the movie. Ifthe user does not wish to control the positioning of the ad movie, atstep 590, transcoder 40 estimates where the movie ad should be placed.Any suitable estimation procedure may be employed. As an example, if thetranscoded movie is a completely new movie with no deep tags, theestimation procedure specifies that the first scene having a length ofat least the length of the ad movie is the selected position, and thatthe ad movie should be inserted as a picture-in-picture in the lowerright of the transcoded movie. However, if the transcoded movie isassociated with at least one other user, then a popularity procedure isused to select the position for the ad movie.

Collection and processing of user statistics will now be discussed.These statistics are employed in the popularity procedure used toautomatically position the ad movie.

FIG. 13 is a flowchart illustrating collection and processing of usagestatistics.

At step 1300, a viewer, such as a user of PC 140, who is understood toalso or alternatively be a listener, requests a movie, such as byclicking on a link to the movie, the link being embedded in an e-mailpreviously sent to the viewer. At step 1325, server 50 receives themovie request, and retrieves the requested movie from videobase 20. Atstep 1330, server 50 sends the movie to the requesting viewer, and atstep 1335, server 50 updates its download statistics for the movie.

At step 1305, the viewer receives the movie, and at step 1310 plays themovie. Playing the movie includes viewing the movie in its intendedtemporal sequence, pausing, replaying a section of the movie,fast-forwarding, adding one or more deep tags and so on. Meanwhile, atstep 1315, a play trail is being recorded. As used herein and in theclaims, a “play trail” means a timestamped sequence of actions that aviewer takes when playing a movie. The timestamp is typically measuredin tenths of a second since the start of the movie, but may be measuredin clock time or by frame number of the movie, or any other convenientmetric. Each action in the sequence carries appropriate information toreconstruct what the viewer did, for example, “pause” indicates how longthe movie was paused. When the viewer finishes viewing the movie, atstep 1320, the play trail is sent to server 50.

FIG. 14 is a timeline illustrating sample play events for a particularviewer. The horizontal access indicates elapsed time in a movie, such asseconds or frames. Events A-E indicate where the viewer replayedportions of the movie. Events F-I indicate where the viewer paused themovie. Events J-K indicate deep tags created by the viewer. A movie playevent corresponds to viewer actions that alter the playing sequence of amovie, that is, make the playing sequence be other than a frame by framedisplay with each frame being displayed for the same time interval.

FIG. 15 is a data structure representing the sample play events of FIG.14, that is, FIG. 15 shows play trail 1400. In general, a play trail hasa head indicating the movie ID, the user ID for the viewer, a datestampfor when the play trail was created, such as when server 50 first storedthe full play trail. The head also serves as the first statistical unit.Each statistical unit points to the next sequential statistical unitcorresponding to the next sequential portion of the movie. Eachstatistical unit also points to a triplet of event type counters: numberof playback events, number of dwell events, and number of deep tags.Each event type counter also points to individual records for theevents. Typically, an individual record records the start time of anevent and its duration; in the case of a deep tag, text for the deep tagmay also be recorded in the individual record. It will be appreciatedthat, when the play trail data is combined with demographic dataassociated with the user ID, a rich set of information is available.

In FIG. 15, head 1410 points to statistical units 1420, 1430 and 1440,having event type counters 1411, 1412, 1413; 1421, 1422, 1423; 1431,1432, 1433; 1441, 1442, 1443, respectively. Event A in FIG. 14 is aplayback of the movie from second 2 to second 7, and is represented inFIG. 15 as a (2, 5), where 5 is the duration, i.e., 7 seconds-2seconds=5. Event A is recorded as a playback event associated with thefirst statistical unit, seconds 0-5 of the movie, and with the secondstatistical unit, seconds 5-10 of the movie, since event A spans bothstatistical units. More specifically, record 1414 and record 1424 bothcorrespond to event A. As another example, event G in FIG. 14 is seen tobe a dwell event at second 14 of the movie; the corresponding record1437 in FIG. 15 indicates a dwell event at second 14 that lasted for 8seconds.

At step 1340 of FIG. 13, server 50 receives the play trail. At step1345, server 50 stores the play trail along with information about whenthe play trail was received, and the identity of the viewer, if known,or the computer address from which the play trail was sent. At step1350, server 50 updates basic statistics for the movie. Basic statisticsinclude, but are not limited to: number of deep tags created for aportion of a movie, access (listening and/or viewing) frequency forportions of a movie, replay frequency for portions of a movie, pausefrequency for portions of a movie, and community score for portions of amovie.

As described above, a play trail is created at the user's location, thensent to server 50. In other embodiments, the play trail is stored in aso-called cookie, a locally stored data file that tracks anauthenticated user's actions on a PC, on the user's PC so that the playtrail data can persist over time in the cookie data even when a user'sbrowser session has completed or has closed prematurely. The cookie canbe extensive in terms of data structures and may include demographicdata as well.

Alternatives to cookies include the use of HTTP authentication, clientside persistence, local stored Flash objects, JavaScript window. namevariables, and so on; each of these methods provides mechanisms to storeand track user play trail data. The advantage of a cookie relative to alocal file is that a cookie uses a standardized Internet mechanism totrack user data in a well-defined location on a user's hard drive. Thedata is tracked across user browser sessions and is persistent acrossthose sessions. Cookies protect user privacy by tracking user behaviorwithout tracking identity. FIG. 15 shows how a play trail can be storedas a data structure; however this same approach can be applied to thestorage of data in a user cookie stored on a PC. An advantage of acookie relative to ongoing HTTP message transmission is that the cookieminimizes network bandwidth during network playback if the cookie datais transferred after the session is complete or during periods ofminimal activity when interacting with a media application.

Play trails are a form of user behavior tracking. Other user behaviorthat is tracked to determine popularity of a movie or movie segmentincludes creating a deep tag and/or sending a deep tag, such as viaemail, instant message (IM), message service such as SMS or MMS, orpodcasting. Deep tags indicate what users consider interesting.

An email or IM version of a deep tag typically comprises a URL providingthe address of the deep tag, such aswww.server.com/deeptag/movielD/StartTime/EndTime, orwww.server.com/movielD/deeptag/deeptag ID. Additionally, the email mayinclude text associated with the deep tag, a thumbnail image for thedeep tag, and an attachment comprising the portion of the movieindicated by the deep tag. SMS, MMS and podcast versions of atransmitted deep tag are similar.

Replay frequency measures how many times a listener/viewer replays aparticular portion, as opposed to its initial playing.

Pause frequency measures how many seconds that a listener/viewer pausesat a particular point in a movie.

Community score is based on ratings provided by listeners/viewers,explained below in connection with FIG. 35. The ratings can be summed,or can be normalized. A normalized score is the total number of starratings by viewers divided by the total number of viewers. For example,a summed score of 500 might occur because 100 viewers gave a 5-starrating to a movie, or because 500 viewers gave a 1-star rating to amovie; in these examples, the normalized ratings would be 500/100=5, and500/500=1.

While FIG. 13 shows a situation in which the entire movie is availableat PC 140 and server 50 is unaware of how the user is interacting withthe movie until server 50 receives the play trail, in other embodiments,server 50 is involved in some user play actions, while PC 140 attends tothe remaining user play actions. For example, playback and deep taggingmay be accommodated at server 50, while pause is accommodated at PC 140.In these other embodiments, server 50 collects portions of the playtrail and PC 140 collects the remainder of the play trail, then PC 140transmits its play trail information to server 50 and server 50 createsa full, combined play trail representing the viewer's actions.

FIG. 16 is a diagram illustrating automatic determination of popularparts of a movie. Horizontal bars 950-955 represent the same movie, asdeep tagged by different users. More generally, popularity can bedefined as a selected metric or as a weighted combination of metrics.The metric can be one of the basic statistics, or a custom-definedstatistic.

Histogram 960, also referred to as popularity density function 960,indicates the popularity metric, herein the number of times that aportion of the movie has appeared in a deep tag, which serves as anindication of the popularity of the various parts of the movie, plottedagainst the sequence in which the frames of the movie are displayed.

Color graph 970 uses different colors to represent different levels ofpopularity. In one scheme, the most popular portions of a movie areindicated by red areas, while the least popular portions of a movie areindicated by blue areas. In another scheme, different shades of gray areused to indicate popularity.

Color graph 970 can be generated as follows. Let the metric beingplotted be the Total Interest in a specific part of the movie. The TotalInterest metric is an indicator of the level of popularity of a sectionof a movie. The Total Interest is a weighted combination of metrics,such as the amount of time paused, the number of playbacks, or thenumber of deep tags created in a specific section of a movie. First, themetric is normalized to be a value between 0-100. Next, the normalizedmetric is mapped to the 0-255 code for each of the display colors ofblue, green, and red.

As a specific example, let the metric plotted be the number of playbackevents per unit time for a movie, and assume the maximum number ofplayback events for any segment of a video is 25, so that 25 playbackevents is normalized to a value of 100. The color code is calculatedusing the following formula:Red=(MAX(((Normalized_Metric−50)*255/50)),0))Blue=(MAX(((50−Normalized Metric)*255/50),0))For: Normalized_Metric<=50: Green=(Normalized_Metric*255/50) else:Green=((100−Normalized_Metric)*255/50)

Sample usage statistics for a particular movie, and their conversion toRGB values are shown in the following table.

Movie no. playback Normalized segment events value Red Blue Green 0 to 5sec 5 20 0 153 102  5 to 10 sec 10 40 0 51 51 10 to 15 sec 0 0 0 255 015 to 20 sec 25 100 255 0 0

The Red Green Blue color values determine the color for a verticalcolumn of pixels in color graph 970 corresponding to the time intervalof the movie segment. As is well-known, each pixel of a display isactually a group of three light emitting elements: red, green and blue,and the intensity values for each of the elements determine the colorperceived by the viewer. In some embodiments, the columns of color graph970 are placed next to each other, like bar graphs having the sameheight, whereas in other embodiments, the colors are interpolated fromcolumn to column, to provide a more aesthetically pleasing appearance.

Cloud graph 980 has clouds or circles whose size corresponds to thepopularity of a movie portion.

Other types of graphical representations of popularity may be employed,such as three dimensional strips with colored segments, threedimensional bar graphs, surface contour maps in three dimensions, and soon.

As used herein and in the claims, popularity statistics are based on oneor more popularity metrics such as total number of requests for a movie,total number of times that the movie has been selected as a favorite(see FIG. 35), total number of times that the movie has been given athree, four or five star rating (see FIG. 35), number of times that aportion of the movie has been played back, number of times that aportion of a movie has been paused by a viewer, total pause time for aportion of a movie, number of times that a viewer has placed a deep tagon a portion of the movie, number of deep tag transmissions relating tothe movie, and so on.

A further aspect of the movie may be provided by demographic or otherselected data about viewers, such as age bracket, gender, location,income level and so on, instead of or in addition to the popularitymetrics discussed above. Of course, this can be accomplished only whenthe viewer provides such demographic data, such as during establishing anew user account.

Popularity graphs can be reported to the owner of the movie.

As shown in FIG. 35, popularity graphs can be displayed to a viewer ofthe movie.

Popularity graphs can also be used internally by transcoder 40. Movieads are placed in the popular parts, if they are long enough, or withrespect to the popular parts, if the popular parts are shorter than thead movie. For example, the movie ad could be placed so that its start isco-located with the start of a popular segment having a duration of atleast 70% of the duration of the ad movie.

Returning to FIG. 13, generation of a custom statistics report, alsoreferred to as a custom popularity report, will now be discussed.

At step 1380, an owner of a movie requests custom statistics about themovie. such as a number of deep tags created by a particular demographicgroup during a particular time period. At step 1355, server 50 receivesthe custom statistics request. At step 1360, server 50 retrievesappropriate play trails, and at step 1365, server 50 retrievesappropriate user data. At step 1370, server 50 generates the requestedcustom statistics report. At step 1375, server 50 sends the customstatistics report to the requester. At step 1385, the requester receivesthe custom statistics report.

Returning to FIG. 10, at step 600, transcoder 40 inserts the selectedmovie ads into the transcoded movie. Ad insertion is shown in moredetail in FIG. 17. Before turning to FIG. 17, the diagrams of FIGS.18A-18C illustrating ad bounceback for static ads, FIGS. 19A-19Cillustrating slip-ad for proxy ads, and FIG. 20 illustrating adbounceback and slip-ad are discussed.

Ad bounceback is a technique for prohibiting skipping of a static ad.The static ad can have a pre-ad zone defined before its start and/or apost-ad zone defined after its end. The pre-ad zone forces a bouncebackto a few frames before the start of the ad; this is useful when thecontext of the movie is helpful to appreciating the ad. The post-ad zoneis relevant to certain types of ads, and prohibits a user's skip frombeginning until after the end of the post-ad zone

FIG. 18A is similar to FIG. 5B that illustrates movie 20B with static ad93B that occupies a portion of the frames of movie 20B. Segment 355indicates frames that are played in a conventional manner. Segment 356indicates that the user has skipped frames, either by fast forwarding orby seeking to a future point in movie 20B. Segment 357 indicatesresumption of conventional frame playback. Segment 356 is seen to skipover ad 93B.

FIG. 18B shows ad bounceback in operation for a single static ad. Ad 93Bhas pre-ad zone 362 and post-ad zone 363. Segment 358 indicates framesthat are played in a conventional manner. Segment 359 indicates that theuser has skipped frames. Segment 360 is a bounceback to the start ofpre-ad zone 362, and segment 361 indicates resumption of conventionalframe playback. Segment 361 is seen to play ad 93B.

FIG. 18C shows ad bounceback in operation for multiple static ads. Ads93E, 93F, 93H are static ads occupying respective portions of respectiveframes of movie 20E. Ad 93G is a static ad occupying the entire framesof movie 20E. Segment 364 indicates frames that are played in aconventional manner. Segment 365 indicates that the user has skippedframes. Since the skip would bypass several ads, namely ads 93E, 93F,93G, 93H, there are different strategies that can be employed todetermine where the ad bounceback goes to. Only one of these strategiesis selected in any embodiment.

One strategy is to bounceback to the earliest ad that would otherwise beskipped. Segment 366 is a bounceback to the start of ad 93E, and segment367 indicates resumption of conventional frame playback.

Another strategy is to bounceback to any ads that the user has indicatedthat he or she wishes to view. For example, prior to viewing a movie,the user may have indicated in preferences provided to movie system 10that he or she is interested in ads relating to washing machines and/orads that pay the user to watch them, via an input mechanism such as auser preferences web page. Assume ad 93F is a washing machine ad.Segment 368 is a bounceback to the start of ad 93F, and segment 369indicates resumption of conventional frame playback.

Another strategy is to bounceback to any ads for which the advertiserhas required, such as by paying a premium, that the ad be unskippable.For example, the advertiser associated with ad 93G may have paid to makead 93G unskippable. Segment 379 is a bounceback to the start of ad 93G,and segment 380 indicates resumption of conventional frame playback.

Another strategy is to bounceback to the ad closest to the target skipdestination, ad 93H in this case. Segment 381 is a bounceback to thestart of ad 93H, and segment 382 indicates resumption of conventionalframe playback.

Slip-ad is a technique for prohibiting skipping of a proxy ad. Moviescan have no-ad zones defined anywhere therein. Slip-ad causes a proxy adto move its position from the skipped material to after the resumptionof playback. In operation, the proxy ad slips to where the user wants towatch.

FIG. 19A corresponds to FIG. 5D, showing original proxy ad 94C thatoccupies a portion of frames of movie 20D. Segment 385 indicates framesthat are played in a conventional manner. Segment 386 indicates that theuser has skipped frames. Segment 387 indicates resumption ofconventional frame playback. Segment 386 is seen to skip over ad 94C.

FIG. 19B shows slip-ad in operation for a single proxy ad in movie 20F.No-ad zone 389 occurs at the end of movie 20F. Segments 385, 386, 387operate as described above. However, original proxy ad 94C slips tomidway between the skip point, that is, the end of segment 386, and thestart of no-ad zone 389. Segment 388 indicates the slipping of originalproxy ad 94C. The destination of original proxy ad 94C is referred to asslipped proxy ad 94C′; the size of the slipped proxy ad can be greateror less than the size of the original proxy ad. Ad 94D is delivered tothe location indicated by slipped proxy ad 94C′, and is part of themovie played to the user.

FIG. 19C shows slip-ad in operation for multiple proxy ads in movie 20G.Movie 20G has original proxy ads 94C, 94E and no-ad zone 389. Segment388 indicates the slipping of original proxy ad 94C to slipped proxy ad94C′. Ads 94D, 94F are respectively delivered to the locations indicatedby slipped proxy ads 94C′, 94E′.

FIG. 20 shows bounceback and slip-ad in operation in the same movie.Movie 20H has static ad 93B, original proxy ad 94C and no-ad zone 389.Segment 391 indicates frames that are played in a conventional manner.Segment 392 indicates that the user has skipped frames. Segment 386 isseen to skip over static ad 94B and original proxy ad 94C. Segment 393is a bounceback to the start of static ad 94B. Segment 394 indicatesresumption of conventional frame playback. Segment 395 indicates theslipping of original proxy ad 94C to slipped proxy ad 94C′, locatedbetween the end of static ad 93B and the start of no-ad zone 389.

FIG. 17 is a flowchart showing details of ad insertion.

At step 602, transcoder 40 of movie system 10 determines whether astatic ad or a proxy ad is being inserted.

If a static ad is being inserted, then at step 603, the ad is insertedinto the appropriate frames of the movie, at the appropriate placewithin the frames. At step 604, it is determined whether this ad shouldbe protected by bounceback, such as by querying the editor. If nobounceback is needed, then processing is complete. If bouncebackprotection is desired, then at step 605, it is determined whether thereshould be a pre-ad zone, and if so, at step 606, how many frames shouldbe in the pre-ad zone. At step 607, it is determined whether thereshould be a post-ad zone, and if so, at step 608, how many frames shouldbe in the post-ad zone. At step 609, the start of the no-seek zone isdefined as the start of the ad minus the pre-ad zone, if any. At step610, the end of the no-seek zone is defined as the end of the ad plusthe post-ad zone, if any. Ad insertion processing is complete.

If a proxy ad is being inserted, then at step 611, the proxy ad isinserted into the appropriate frames of the movie, at the appropriateplace within the frames. At step 612, it is determined whether this adshould be protected by slip-ad, such as by querying the editor. If noslip-ad is needed, then processing is complete. If slip-ad protection isdesired, then at step 613, it is determined whether there should be atleast one no-ad zone in the movie, and if so, at step 614, the framesineligible to receive a slipped proxy ad are indicated. At step 615, itis determined whether the size of the slipped proxy ad should beadjusted relative to the size of the original proxy ad. If so, at step616, the number of frames in the slipped ad proxy are defined, and aninsertion processing is complete. An increased size basically provides abuffer zone relative to any other ads in the movie. A decreased size ismeaningful when the proxy ad is a graphic image, and increases thechance that the slipped ad proxy will be small enough to fit into theunskipped part of the movie. If the slipped ad proxy does not requiresize adjustment, then at step 617, the size of the slipped ad proxy isset equal to the size of the original ad proxy, and ad insertionprocessing is complete.

FIGS. 21-23 are flowcharts illustrating different techniques for usingpopularity statistics to insert ads in a movie. FIG. 21 shows a one-timead insertion procedure. FIG. 22 shows a periodic ad insertion procedure.FIG. 23 shows a real-time, dynamic, “on the fly” ad insertion procedure.

FIG. 21 shows the technique just described, wherein transcoder 40inserts ads into a movie. At step 1505, transcoder 40 obtains the movie,such as because the movie is being edited. At step 1510, transcoder 40determines whether the movie has been viewed.

If the movie has been previously viewed, at step 1515, transcoder 40obtains movie characteristics and popularity statistics for the movie,such as popularity density graph 960, color graph 970, cloud graph 980,or a computed metric such as Total Interest, discussed above as aweighted sum of various play events. Transcoder 40 also obtains theplacement rules for an ad. Typically, an ad provider specifies defaultplacement rules when their account is set up, and can modify, oroverride for a particular ad or movie, these default placement rules asthey wish. Default ad provider placement rules specify whether an admust be placed manually or automatically, by transcoder 40. Ifautomatically, the ad provider placement rules are able to furtherspecify whether the ad should be aligned with the start of a popularportion, be centered on a popular portion, be aligned with the end of apopular portion, be placed in the popular portion of closest duration tothe ad, or other placement procedure. The duration of the ad and theduration of the movie may or may not be identical. The ad providerplacement rules are also able to specify whether the ad providerrequires the most popular portion of the movie, or will accept anotherportion that is still popular, such as by indicating how much of apremium the ad provider is willing to pay. In some embodiments, foraesthetic reasons, transcoder 40 generally ensures that there is onlyone movie ad in any portion of a movie.

On the other hand, if the movie has never been viewed, which in someembodiments corresponds to the number of views being below a threshold,then transcoder 40 uses a predefined procedure based on moviecharacteristics to place ads. Instances of a predefined procedure for adplacement include: (i) to place ads so they are uniformly distributed intime throughout the video, (ii) clustering ads into groups between movieportions of two to three minutes duration, permitting four to six ads tobe inserted for every eight to ten minutes of movie content, as is donein broadcast television, (iii) to place ads at scene changes in themovie, (iv) to place ads at mid-scene in the movie, and so on. In someembodiments, for aesthetic reasons, transcoder 40 generally ensures thatthere is only one movie ad in any portion of a movie.

At step 1525, transcoder 40 places an ad in the movie, with theplacement determined either by popularity statistics or the predefinedprocedure. At step 1530, transcoder 40 stores the movie with ad(s) invideobase 20.

FIG. 22 is similar to FIG. 21, except that in FIG. 22, a timerconstantly runs and at a specified period, such as every hour, everyday, every week, or other appropriate period, at step 1535, transcoder40 redetermines the position of the ad in the movie.

FIG. 23 shows real-time ad insertion. At step 1570, a potential viewerrequests a movie. At step 1572, transcoder 40 gets the characteristicsof the viewer, if possible. For example, when a user registers withmovie system 10, the user can specify their demographics; if thepotential viewer is a registered user, then their demographics areavailable. At step 1575, transcoder 40 obtains the movie, and asdescribed above, inserts ads using either popularity statistics or apredefined ad insertion procedure; here, the characteristics of theviewer may also be used to determine what ad(s) to place in the movie.At step 1597, transcoder 40 sends the movie to the viewer.

Other procedures may be used to automatically insert an ad into a movie,instead of or in combination with the techniques described above.

Motion analysis of a movie may be used to determine where to place anad, for example, in a scene having the least amount of motion, oralternatively in a scene having a lot of motion but in a relativelystatic area of the high motion scene. Motion analysis ensures that adplacement is complementary to activity in the movie, rather thanobliterating it. For example, some advertisers may specify that their adshould be placed so that it does not interfere or intersect withdramatic scene changes or scenes of high interest with primarycharacters visible in the scene.

Face analysis of a movie may be used to ensure that an ad is not placedover a face and/or to ensure that an ad is placed in a scene with atleast one human face. Advertiser business rules may indicate apreference for ad placement near faces (i.e., just above or below) sothat they do not obscure any characters but are assured that theiradvertisement will be in an important scene.

Image analysis of a movie may be used to ensure that text or logos arenot obscured by an ad. Alternatively, it may be desirable to obscure alogo in the movie with an ad.

Examples of products that perform object recognition and face detectionon a video stream are:

-   -   Autonomy Virage Smart Encode, available at        http://www.virage.com/content/products/vs-smartencode/index.html,        and    -   Intel Open Source Computer Vision tools, available at        http://www.intel.com/research/mrl/research/opencv/

FIG. 24 shows ad editor interface 1100 having ad area 1110, adpack area1121 and texture strip 1130.

Ad area 1110 includes ad thumbnail windows 1111-1120 and navigationbuttons 1101, 1102 for altering which thumbnails of the ad thumbnails invideobase 20 are displayed in area 1110. Filename entry window 1103enables a user to type in the name of an ad file, or select an adfilename from a directory, which puts the file's thumbnail in adthumbnail window 1111. Ad area 1110 also includes automatic button 1104,for indicating that movie system 10 should select an ad.

An adpack is a pairing of an ad movie and an adpack description. Anadpack description controls how the ad movie will be displayed in thetranscoded movie. Examples of adpack descriptions are:

-   -   Ad image at frame bottom    -   Ad image at frame bottom, fade in and fade out    -   Ad image slides from left to right in succeeding frames    -   Ad audio plays when frames are viewed    -   Ad video at frame top    -   Ad video at frame bottom    -   Ad video in frame left    -   Ad video in frame right

The International Advertising Bureau has defined standard ad displayareas, and these may be used as adpack descriptions; see, for example,http://www.iab.net/standards/popup/index.asp.

An adpack can also include one or more executable commands, as describedabove with respect to the tree playback product editor. The provider ofan ad movie typically specifies certain adpacks as selectable for its admovie, and configures the commands to be associated with its ad, movie.Examples of commands are: (i) a hyperlink to the ad provider's web site,(ii) sending the email address of the viewer of the transcoded movie tothe ad provider, and (iii) requesting a file download from the adprovider to the viewer of the transcoded movie; other commands are alsocontemplated.

If the ad is a proxy ad, the adpack includes the address of the proxy adserver where the ad resides.

An adpack can also include special effects and scripts. For example, anadpack may be comprised of a special effect to fade in a graphicalbanner, then to fade out the graphical banner, wherein the graphicalbanner is overlaid in a video stream that permits the user to execute ascript when clicking on the banner ad, the script causing the user'sbrowser to launch a web site for an advertiser with analytics andtracking for the user interactions.

Adpack area 1121 includes adpack description windows 1125-1128 andnavigation buttons 1122, 1123 for altering which adpack descriptions aredisplayed in area 1121. Filename entry window 1124 enables a user totype in the name of an adpack, or select an adpack from a directory,which puts the adpack in adpack description window 1125. Adpack area1121 includes script parameters button 1129, for enabling a user toenter parameters for a script. Adpack area 1121 also includes automaticbutton 1105, for indicating that movie system 10 should select theplacement of the ad in the frame.

Texture strip 1130 includes frame representations of the movie beingedited, such as frame representation 1131. Positioner 1132 enables rapidselection of a frame of the movie. Slider 1135 indicates framerepresentations in the texture strip; the editor can adjust the left andright edges of slider 1135. Automatic button 1106 is used when theeditor wishes to indicate that movie system 10 should select the framesin which the ad is placed.

To manually insert an ad, the editor selects an ad, such as by clickingad thumbnail 1117, then selects an adpack description to control wherein the frame the ad is placed, such as by clicking adpack description1127, then adjusts slider 1135 to indicate which frames the ad is placedin, then clicks insert button 1133.

To instruct movie system 10 to select an ad and put it in the moviebeing edited, the editor clicks automatic buttons 1104, 1105 and 1106.

Rules button 1107 enables the editor to specify the advertiser's adplacement preferences. Advertiser preferences can specify moviecharacteristics, movie popularity characteristics, viewercharacteristics, and how much the advertiser is willing to pay dependingon the combination of features delivered, which may include whether thead is viewed or whether the viewer takes an action relating to the ad.For example, some ads include hyperlinks that a viewer can click on, andthe advertiser may be willing to pay a first rate if the ad is merelypresented to the viewer, and a second rate if the viewer clicks on thehyperlink. Clicking rules button 1107 makes another window (not shown)appear, that guides the editor through specifying the advertiserpreferences. In some embodiments, when an advertiser first registerswith movie system 10, the advertiser specifies their defaultpreferences; in these embodiments, the window (not shown) has thedefaults filled in, and enables the editor to override the defaults.

To remove an ad, the editor adjusts slider 1135 then clicks cut button1134.

Buttons 896, 897, 898, 899 function as follows.

Preview button 896 being actuated causes transcoder 40 creates previewwindow 802, shown in FIG. 28, having viewing area 803, and navigationbutton 804-807, and begins playing the mash-up in preview window 803.The user clicks back button 804 to “rewind” the playback, forward button805 to “advance” the playback, pause button 806 to pause the playback,and stop button 807 to terminate the playback and close preview window802.

Publish button 897 being actuated causes transcoder 40 to pop-up awindow (not shown) that enables the user to select a thumbnail anddestination.

Burn button 898 enables the user to indicate to transcoder 40 thatburning (recording) onto a medium is desired. Clicking burn button 898causes transcoder 40 to pop-up a window (not shown) that enables theuser to select a media for burning and provide delivery directions forthe burned media.

Back button 899 enables the user to return to edit window 700 in FIG.11.

Returning to FIG. 10, at step 620, transcoder 40 determines whether theuser wishes to share the transcoded movie, or segments thereof. Forexample, the user indicates a desire to share by selecting share button790 in FIG. 11. If so, at step 625, the user identifies segments by deeptagging them, or using previously defined deep tags. At step 630, theuser identifies who is permitted to access which segments. In someembodiments, when a user registers with movie system 10, the user cancreate access groups having one or more outside users in each group; thesame outside user can be included in one or more access groups. Anoutside user is generally identified by a nickname and an email address.Then, the user instructs transcoder 40 when to make the accesspermissions effective, typically by sending an email to the outsideusers identifying the segment address, and its short text description.The segment address is of the form: (server 50 address, transcoded movieidentifier, deep tag identifier). In some embodiments, the user can, aspart of sharing, list the transcoded movie in a directory accessible toother users of movie system 10, searchable by characteristics of thetranscoded movie, such as its metadata subject keywords, its creationtime, and/or its ads.

FIG. 25 is a diagram illustrating access permission for a movie. Let itbe assumed that the owner of movie 900, user1, defines a deep tagindicating segment 904 of movie 900, and permits an outside user, user2,to access segment 904. Now, in movie 900, segments 902 and 906 areprivate segments, while segment 904 is a shared segment.

User2 sees segment 904 as movie 910. User2 can define a deep tagindicating segment 914 of movie 910, and permit an outside user, user3,to access segment 914. Now, in movie 910, segments 912 and 916 areprivate to user2, while segment 914 is shared.

User3 sees segment 914 as movie 920, and can similarly enable access toall or parts of movie 920.

At step 631 of FIG. 10, transcoder 40 determines whether the user wishesto create a playback product. If so, at step 632 of FIG. 10, the stepsof FIG. 26 are executed by transcoder 40.

If the movie that is to be turned into a playback product includes oneor more static ads, then the playback product includes those ads andviewers of the playback product see the static ads. If the movieincludes one or more dynamic ads, then the playback product includesthose dynamic ads; if the player of the playback product is suitablycoupled to a communication network and can interpret the embeddedinformation to dynamically retrieve ads, then the viewer of the playbackproduct will see the dynamic ads, otherwise, the viewer of the playbackproduct will not see the dynamic ads.

Turning to FIG. 26, at step 650, transcoder 40 receives the user'sselection of which playback product is to be created. For instance,using the editing interface of FIG. 11, the user indicates whichplayback product by selecting one of buttons 771, 772, 773. In thisembodiment, three products are defined. In other embodiments, otherproducts are available. The playback products are a mash-up, a tree, anda link-up.

A mash-up is a sequential display of selected segments. A viewer of amash-up playback product can only navigate forward or backward in theproduct.

A tree is a set of segments and a hierarchical, linear control structurefor displaying the segments. Generally, a viewer of a tree playbackproduct clicks on selections to navigate the product, in addition toforward and backward.

A link-up is a set of segments and a non-linear control structure fordisplaying the segments. Generally, a viewer of a link-up playbackproduct navigates via one or more of: forward and back movement,clicking on selections, and/or providing alphanumeric input. A tree anda mash-up are constrained forms of a link-up.

If a mash-up is selected, at step 655, the user selects the sequence ofsegments to be included via a mash-up editor. Then, at step 685, theuser selects whether the mash-up is to be published or burned.

Publishing means transferring the mash-up to videobase 20 or to user PC110. If publishing is selected, at step 690, the user selects athumbnail to represent the mash-up, and provides metadata if desiredsuch as a mash-up filename. At step 695, the user indicates adestination for the mash-up file, such as videobase 20, or PC 110.Transcoder 40 responds by transferring the mash-up in accordance withthe user's selections.

Burning means writing the mash-up to a removable storage medium, such asa DVD or memory chip, and sending the removable storage medium to theuser. If burning is selected, at step 698, transcoder 40 transfers themash-up file to the removable storage medium type designated by theuser. In the case of a DVD, transcoder 40 sends the mash-up file to DVDburner 60, which creates a DVD having the mash-up.

FIG. 27 shows mash-up editor interface 801. Thumbnail area 810 includesnavigation buttons 825, 826 for altering which thumbnails of the user'sstored thumbnails are displayed in area 810. In general, a user acquiresstored files, represented by thumbnails, by uploading a movie, image,graphic or audio file; selecting frames of an existing movie; orcreating a playback product as described herein. After storing a file,the user can edit the file by selecting the file's thumbnail in one ofthe specialized editors discussed herein. Thumbnail area 810 shows tenthumbnail windows 811-820, but a user may have hundreds of stored files,each with a corresponding thumbnail. Filename entry window 827 enables auser to type in the name of a file, or select a filename from adirectory, which puts the file's thumbnail in thumbnail window 811.

Slots area 836 comprises placeholders into which the editor, alsoreferred to as the user, drags and drops thumbnails to indicate that thethumbnails are part of the mash-up being created. Slots area 836includes slots 837, 838, 839, 840.

Texture strip 830 represents the mash-up being created. Phantom startand end frames 831, 832 enable the user to add thumbnails before orafter the selected thumbnails. Frame selector 835 has start and endportions that can be adjusted by the user. After the user is satisfiedwith the thumbnails dragged into slots 837-840, the user clicks insertbutton 833 to insert these thumbnails into the mash-up. In response,transcoder 40 creates a frame representation of each thumbnail, puts thethumbnail frame representation in the appropriate frame of texture strip830, and clears slots 837-840. To insert subsequent files into themash-up, the user moves frame selector 835 to a position after theinserted thumbnails. To insert preceding files into the mash-up, theuser moves frame selector 835 to include phantom frame 831. To deletefiles from the mash-up, the user positions frame selector 835 on theframe representations of the thumbnails of the files to be deleted, andclicks cut button 834.

At any time, the user can click preview button 896 to see what themash-up will look like. In response to preview button 896, transcoder 40creates preview window 802, shown in FIG. 28, having viewing area 803,and navigation button 804-807, and begins playing the mash-up in previewwindow 803. The user clicks back button 804 to “rewind” the playback,forward button 805 to “advance” the playback, pause button 806 to pausethe playback, and stop button 807 to terminate the playback and closepreview window 802.

Publish button 897 enables the user to indicate to transcoder 40 thatpublication of the mash-up is desired. Clicking publish button 897causes transcoder 40 to pop-up a window (not shown) that enables theuser to select a thumbnail and destination.

Burn button 898 enables the user to indicate to transcoder 40 thatburning of the mash-up is desired. Clicking burn button 898 causestranscoder 40 to pop-up a window (not shown) that enables the user toselect a media for burning and provide delivery directions for theburned media.

Back button 899 enables the user to return to edit window 700 in FIG.11.

If a tree playback product is selected, at step 660, the user selectsthe sequence of segments to be included via a tree editor, and at step665, defines the tree structure.

Then, at step 685, the user selects whether the tree playback product isto be published or burned.

FIG. 29 shows tree editor interface 851. Tree editor interface 851 isgenerally similar to mash-up editor interface 801; similarities will notbe discussed for brevity.

Tree structure window 855 shows a graph of the control structure of thetree playback product being created. Initially, the window is blank,since nothing has been specified.

FIG. 30 shows an example of tree structure window 855 after a treeplayback product has been created. The tree playback product includessegments 855-1, 855-2, 855-3 and 855-4. Window 855 shows, via lines, thecontrol structure for the navigation between the segments of theplayback product. In this example, segment 855-1 leads to segment 855-2which in turn leads to either of segment 855-3 and 855-4.

Each segment of a tree playback product comprises a background, aforeground and a link. In one embodiment, a background is a still image,a foreground is a video or audio segment, and the link is indicated by agraphic image. When the viewer of the playback product clicks on thelink, the viewer is taken to the next segment, that is, a link indicatesone segment. Each segment can have 0, 1 or multiple links.

A tree segment can also include 0, 1 or multiple commands. Typically, acommand is indicated by a graphic image. When the viewer of the playbackproduct clicks on the command, the command is sent to the source of theplayback product, such as server 50, for execution.

Returning to FIG. 29, to create a segment, the editor drags and dropsthumbnails from thumbnail window 810 into at least one of backgroundslot 861, foreground slot 862, and link slot 863. If more links aredesired for this segment, the editor clicks add link button 864, andthen drags and drops thumbnails into the additional link slots (notshown) created by clicking link button 864. When the user is satisfied,he or she clicks insert button 867 to create a segment.

If a tree playback product segment is created with at least one link,transcoder 40 creates an empty segment as the destination of each link,and displays the empty segment in tree structure window 855. The editorclicks on the empty segment in tree structure window 855 and insertsthumbnails into at least one of the background, foreground and linkslots.

If the editor wishes to delete a segment, the editor selects the segmentin tree structure window 855, then clicks cut button 856 to remove thesegment. Removing a segment automatically removes the link leading tothe segment from the preceding segment.

To create a command in a segment, the editor clicks add command button865. Transcoder 40 provides a pop-up window with a command editor (notshown) that enables the editor to drag and drop a thumbnail indicatingthe command, select a command from a menu of commands or type thecommand directly into a command line window (if the editor knows how towrite commands in the command language, such as Javascript), and, whenappropriate, provide parameters for the command. Examples of commandsare: (i) a hyperlink to a webpage, (ii) provide the email address of theviewer of the playback product to the owner of the playback product,(iii) provide the email address of the viewer to a third party, (iv)download a program and execute it, and so on.

It will be understood that a thumbnail can be dropped into multipleslots during creation of a tree playback product.

Clicking preview button 896 causes transcoder 40 to create a windowsimilar to that shown in FIG. 28, however, the tree playback productincludes link areas in window 803 that a user can click to navigatebetween segments.

If a link-up playback product is selected, at step 670, the user selectsthe sequence of segments to be included via a link-up editor. At step675, the user defines the link-up structure. At step 680, the userdefines the link-up navigation questions and answers. Then, at step 685,the user selects whether the link-up is to be published or burned.

FIG. 31 shows link-up editor interface 871. Link-up editor interface 871is generally similar to tree editor interface 851; similarities will notbe discussed for brevity.

Link-up structure window 875 shows a graph of the control structure ofthe link-up playback product being created. Initially, the window isblank, since nothing has been specified.

FIG. 32 shows an example of link-up structure window 875 after a link-upplayback product has been created. The link-up playback product includessegments 875 a, 875 b, 875 c and 875 d. Window 875 shows, via lines, thecontrol structure for the navigation between the segments of theplayback product. In this example, segment 875 a, always the firstsegment to be played during playback, leads to segments 875 b, 875 c and875 d. Segment 875 b leads to 875 a and 875 c. Segment 875 c leads tosegments 875 b and 875 d. Segment 875 d is sometimes referred to as afinal segment, since it is never followed by another segment. Segments875 b and 875 c are sometimes referred to as intermediate segments,since they are not the start segment and they do lead to anothersegment.

Returning to FIG. 31, to create the link-up structure, the editor dragsand drops thumbnails from thumbnail window 810 into start slot 880 andat least one of intermediate slots 881, 882, 883. If more links aredesired for this segment, the editor clicks add link button 878, andthen drags and drops thumbnails into the additional slots created byclicking link button 878 (not shown). When the user is satisfied, he orshe clicks insert button 877 to create the link-up. Transcoder 40responds by displaying the link-up in link-up structure window 875,highlights the just-created bit of link-up structure, and freezes thehighlight until the editor clicks edit Q&A button 879 to create textassociated with the link-up.

A Q&A dialog (not shown), consists of an optional header and a set ofchoices associated with the respective links. One instance of a Q&A is,

-   -   Click on the word to indicate what you want to view next:        -   kitchen        -   den        -   basement        -   garage        -   backyard    -   Another instance of a Q&A is,        -   More pictures of actor Alice        -   More pictures of actor Bob        -   More pictures of actor Chris    -   Another instance of a Q&A is,        -   Type (or click on) the number closest to your age:            -   15            -   20            -   25            -   30

If the editor wishes to delete a segment, the editor selects the segmentin link-up structure window 875, then clicks cut button 876 to removethe segment. Removing a segment automatically removes the portion of theQ&A leading to the segment from the preceding segments.

Clicking preview button 896 causes transcoder 40 to create a windowsimilar to that shown in FIG. 28, however, the link-up playback productincludes Q&A areas in window 803 that a viewer can utilize to navigatebetween segments.

The transcoded movie is then stored in videobase 20, and database 30 issuitably updated.

A use case will now be discussed.

Let it be assumed that a user uploads three files (FIG. 10, step 400).The first file is a DVD file in MPEG2 format. The second file is awebcam file in MPEG4 format. The third file is a mobile phone camerarecording in 3GP format. At this time, the MPEG2, MPEG4 and 3GP formatsare considered incompatible.

Movie system 10 converts each of these three files to proxy files inH.263 Flash format (FIG. 10, step 420). Let it be assumed that each ofthe proxy files has 100 frames, and have the filenames:

-   -   Proxy-1200    -   Proxy-1201    -   Proxy-1202

Now the user creates a mash-up (FIG. 10, step 450) using the mash-upeditor (FIG. 27). Transcoder 40 represents the mash-up as a list of theproxy files and the start and end frames of each proxy file:

-   -   Proxy-1200, start 22, end 27    -   Proxy-1201, start 84, end 86    -   Proxy-1202, start 62, end 70

Next, the user adds special effects (FIG. 10, step 480) using theeffects editor (FIG. 12). Let it be assumed that one special effect is awatermark (effect 45) comprising an image in the user's file proxy-678,and another special effect is a dissolve (effect 18) between the seventhto ninth frames of the mash-up. Transcoder 40 represents the result as:

-   -   Proxy-1200, start 22, end 27    -   Proxy-1201, start 84, end 86    -   Proxy-1202, start 62, end 70    -   Effect 45, proxy-678    -   Effect 18, start 7, end 9

Finally, the user inserts two ads, allowing the system to automaticallychoose the ads and their placement (FIG. 10, steps 530, 560, 590) usingthe ad editor (FIG. 24).

Transcoder 40 represents the result as:

-   -   Proxy-1200, start 22, end 27    -   Proxy-1201, start 84, end 86    -   Proxy-1202, start 62, end 70    -   Effect 45, proxy-678    -   Effect 18, start 7, end 9    -   Ad 4511268, adpack 2, start 3, end 6    -   Ad 3897522, adpack 163, start 11, end 16

FIG. 33 shows the texture strip resulting from this use case. Texturestrip 1200 comprises frame representations 1201-1218. As indicated bybracket 1271, frame representations 1201-1207 are from file Proxy-1200.As indicated by bracket 1272, frame representations 1208-1210 are fromfile Proxy-1201, As indicated by bracket 1272, frame representations1211-1218 are from file Proxy-1202. Watermark 1230, the logo in fileproxy-678, is present in the bottom right of each frame, as depicted ineach of the frame representations. Dissolve special effect 1260 ispresent in frame representations 1207-1209. First ad 1240, having framerepresentations 1241-1244, in present as a top banner in frames1203-1206. Second ad 150, having frame representations 1251-1256, ispresent as a side banner in frames 1211-1216.

It will be appreciated that some editing functions can be accomplishedin several ways. For example, if the user wishes to delete a frame fromthe movie, this can be accomplished via (1) creating a mash-upcomprising the frames before the deleted frame, followed by the framesafter the deleted frame, and not including the deleted frame; (2) addinga special effect to cause the deleted frame to be entirely dark orentirely light, as part of a scene transition; (3) directly selecting“delete frame” from the special effects menu; (3) enabling sharing ofthe entirety of the movie except for the to-be-deleted frame; and so on.

FIG. 34 is a partial data schema for database 30. Generally, database 30can be viewed by users or by movies having appropriate accesspermission. The schema is structured in accordance with usage of moviesystem 10.

FIG. 35 shows screen display 1390, provided, for example, at PC 140 orphone 130, and including video display 1391, texture strip 720,positioner 730, deep tag marker 735, deep tag thumbnails 738A-738C, deeptag button 740, navigation buttons 804-807, send button 1392, favoritesbutton 1393, popularity selector 1395, rating selector 1396 and grapharea 1394. Screen display 1390 is typically used by a listener/viewer ofa movie.

Elements 720, 730, 740 and 804-807 function as described above; thedescription will not be repeated here for brevity.

Deep tag thumbnails 738A, 738B, 738C appear in a deep tag windowpane.Each of the deep tags associated with the movie being displayed has anassociated thumbnail. Each thumbnail is a small image and associatedtext including deep tag creator name, creation date and time, and anykeywords provided by the creator. The small image is configured by thedeep tag creator, and can be a portion of an image in a scene, ashrunken image of the scene, an image of the creator, or other suitablevisual cue. The deep tag windowpane enables the viewer to scroll amongthe deep tag thumbnails for the movie. Typically, the thumbnails appearin sequential order corresponding to their location in the movie, butother orders can be used, such as popularity order.

Send button 1392 is used to email a hyperlink of the movie to an emailaccount. Clicking on send button 1392 causes a window to pop-up forentry of an email address, similar to the pop-up window shown in FIG.42. The pop-up window may have a drop down menu of previously used emailaddresses arranged by frequency of use, or alphabetical order and so on.In some embodiments, the window enables provision of a text messageaccompanying the hyperlink.

Favorites button 1393 is used to add the movie to the viewer's list offavorite movies maintained by the player for the user.

Popularity selector 1395 enables the user to control display of apopularity graph, as explained above with regard to FIG. 13. Popularityselector 1395 comprises a set of so-called radio buttons, from which onebutton can be selected. Selecting “None” suppresses display of apopularity graph. In FIG. 35, “Color graph” is selected, and so colorgraph 970 of FIG. 13, as appropriate for the movie being viewed, isprovided in graph area 1394. In other embodiments, popularity selector1395 uses a different format, such as a drop-down menu instead of radiobuttons.

Rating selector 1396 enables the user to rate the movie. Rating selector1396 comprises a set of so-called radio buttons, from which one buttoncan be selected. The default is “unrated”. If a viewer wishes to ratethe movie, she or he clicks on the appropriate number of stars, from onestar to five stars. In other embodiments, rating selector 1396 uses adifferent format, such as a slider bar instead of radio buttons.

FIG. 36 is a configuration diagram for another embodiment of the presentinvention depicting the entities involved in deep tag usage. FIG. 36shows movie system 10 as including server 50, network 70, known contentowner movie-id-prints database 31, known content owner deep tagsdatabase 32, unknown content owner movie-id-prints database 33, andunknown content owner deep tags database 34. FIG. 36 also shows server80, movie storage 90, communication network 100, PC 140, phone 131, andregistered content owner host 141.

Server 80 serves to provide a website to visitors that access thewebsite using network 100. Typically, server 80 provides software frommovie system 10 that enables display of movies stored in movie storage90 using the screen interface shown in FIG. 41. In some cases, theprovided software operates to send a so-called browser plug-in programto a browser used by a visitor to the website provided by server 80.

Phone 131 is able to communicate wirelessly with communication network100, and to play movies, receive email and so on.

PC 140 is a general purpose computer executing software in accordancewith the present invention. In one embodiment, PC 140 has a web browserthat is capable of interacting with information received from server 50or 80.

Registered content owner host 141 is a general purpose computerprogrammed to receive and send messages using communication network 100,and may be a personal computer or a web server coupled to othercomputers.

It will be recalled that a deep tag can be thought of as a movie segmentbookmark.

A movie-id-print is like a fingerprint for a movie, that is, a short wayto identify a movie.

It is recognized that the ability to deep tag movies, and to send thedeep tags to others opens the door to copyright concerns. Theconfiguration shown in FIG. 36 addresses this problem by enabling moviecontent owners to specify rules for deep tag creation, and further triesto address copyright concerns by tracking deep tagging of movies havingunknown content owners, notifying registered content owners of deeptagging of a movie that has an unknown content owner, and, afterownership is established, ensuring all deep tags comply with the contentowner's rules. These features are discussed in detail below.

FIG. 37 is a flowchart showing creation of a deep tag and sending thedeep tag to another party.

At step 2070, the user of PC 140 views a movie, either from moviestorage 20 that is part of movie system 10 (see FIG. 1), or from moviestorage 90. At step 2072, the user indicates that he or she wishes todeep tag a portion of the movie, such as by clicking on deep tag button740 or fast deep tag button 741 (see FIG. 41, below). See the discussionbelow of different ways that a fast deep tag button can indicate whatframes should be deep tagged. At step 2074, the user optionally providestext to accompany the deep tag. At step 2076, the user optionally setsaccess permission for the deep tag. At step 2078, the user specifies theemail address, or so-called instant message id, of one or morerecipients of the deep tag. At step 2080, the user sends the deep tag tothe named recipient, such as the user of phone 131.

At step 2082, the user of phone 131 receives the deep tag, such as inthe form of a hyperlink in an email, an icon in an instant messagingsystem, or other suitable method. At step 2084, the user of phone 131activates the deep tag, which causes a movie player to begin executingon phone 131 and to play the movie segment indicated by the deep tag.

FIGS. 38A and 38B show embodiments of deep tags. FIG. 38A corresponds tothe deep tag information shown in FIG. 34, namely:

-   -   deep tag ID    -   movie ID (for the movie the deep tag is associated with)    -   user ID (the creator of the deep tag)    -   deep tag start time relative to the movie    -   deep tag end time relative to the movie    -   a deep tag text description provided by the user    -   number of deep tag snapshots (thumbnails) associated with the        deep tag    -   pointers to the deep tag snapshots

It will be recalled that a snapshot, also referred to as a thumbnail, isa frame chosen to represent the object, for example, a frame at thestart, middle or end of the deep tag.

FIG. 38B includes the information of FIG. 38A plus the following:movie-id-print, for convenience in converting from an unknown contentowner to a known content owner, the creation date and time of the deeptag, and for each time the deep tag is sent: the sender address, therecipient address, the date and time that the recipient first viewed thedeep tagged information, and a text comment, if any, provided by therecipient; an address can be an email address, an instant message ID,and IP address or any other suitable identifier.

Deep tags for which the content owner is known are stored in knowncontent owner deep tags database 32. Deep tags for which the contentowner is unknown are stored in unknown content owner deep tags database34.

FIG. 39 depicts the process of converting movie 20J into texture strip720. As explained above, transcoder 40 builds the texture striprepresenting the movie by applying a function, shown as “f(.)” in FIG.39, to each frame to generate texture data. Transcoder 40 is typicallybuilt so that the function is a module, sometimes selected from alibrary of modules. Texture strip 720 may represent each frame of movie20J, or may represent selected frames of movie 20J, with the selectionbeing periodic, random, or according to another predetermined procedure.For example, the function might be to extract the center 8.times.8pixels of each frame and realign into a 64 pixel height column and thetexture strip is the sequence of 64 pixel columns. In some embodiments,transcoder 40 checks whether the center set of 8.times.8 pixels aresubstantially unchanged from frame to frame, and if so, selectsdifferent pixels from each frame such as sweeping the position of the8.times.8 pixel set along a diagonal, from frame to frame, or othersuitable technique.

FIGS. 40A-40D depict different ways to create a movie-id-print.Generally, a function g(.) is applied to something to generate themovie-id-print. In the technique of FIG. 40A, the function g1(.) is theidentity function, so that texture strip 720 is the same asmovie-id-print 2200. In the technique of FIG. 40B, the function g2(.) isa subset selector, so a portion of texture strip 720 is used asmovie-id-print 2210. FIG. 40B is a more detailed view of texture strip720, corresponding to when texture strip 720 comprises columns ofpixels. In the technique of FIG. 40C, the function g3(.) is a hashfunction and is applied to texture strip 720 to create movie-id-print2220. In the technique of FIG. 40D, the function g4(.) is a hashfunction and is applied to movie 20J to create movie-id-print 2230.

Movie-id-prints for which the content owner is known are stored in knowncontent owner movie-id-prints database 31. Movie-id-prints for which thecontent owner is unknown are stored in unknown content ownermovie-id-prints database 33.

FIG. 41 is a diagram of a user interface for playing a movie, and issimilar to the diagram of FIG. 35. Corresponding elements will not bedescribed for brevity. It will be recalled that the typical way tocreate a deep tag is by adjusting the left and right sides of deep tagmarker 735, such as by dragging, then clicking deep tag button 740. Ifdesired the thus-created deep tag can be sent to another party usingsend button 1392. If no deep tag has been indicated, send button 1392will select the address of the entire movie being viewed fortransmission to another party.

Deep tag imprint 2320 is superimposed on texture strip 720, andindicates frames that have a deep tag associated therewith. There may bemany instances of deep tag imprint 2320 for each texture strip 720, eachinstance corresponding to a separate deep tag, the deep tag being storedin known content owner deep tag database 32 or unknown content ownerdeep tag database 34.

Deep tag descriptor 2330 is associated with deep tag imprint 2320, andprovides the text descriptor, if any, that is part of the deep tag. Forexample, if a cursor is positioned on deep tag descriptor 2330, a smallarea opens on screen 2300 and displays the text descriptor. Alternately,a pop-up window displays the text descriptor.

Deep tag thumbnail 2335 is similar to deep tag thumbnail 738A of FIG.35, except that deep tag thumbnail 2335 is superimposed on the moviedisplayed in viewing area 2310 when the movie frames indicated by thedeep tag are displayed. In practice, deep tag thumbnail 2335 may beprovided as a composite overlay that is translucent, so that the movieframe of viewing are 2310 is visible “beneath” deep tag thumbnail 2335.When there are plural deep tags associated with a movie frame, thenplural deep tag thumbnails appear in viewing area 2310 while the movieframe is displayed.

Deep tag send button 2340 is associated with deep tag imprint 2320, andprovides a quick way to send the deep tag to another party. For example,if the cursor clicks on deep tag send button 2340, a small pop-up windowopens and enables the user to specify the recipient's address.

A streamlined way for creating and sending a deep tag is also provided,sometimes referred to as a “one-click deep tag”.

Assume that movie 20J is being viewed in screen display 2300 on PC 140by a user. Movie 20J is typically provided from movie storage 20 (FIG.1), but can alternatively be provided from movie storage 90 (FIG. 1 or36). When the user clicks on fast deep tag button 741, a deep tag isautomatically created and pop-up window 2350, shown in FIG. 42, appearsso that the user can specify where, if anywhere, the deep tag should besent.

Generally, the owner of the website from which movie 20J is providedspecifies how a fast deep tag is created, selecting from one or more ofthe following:

-   -   User preferences—follow the preference of a registered (formal)        viewer for how the fast deep tag button operates, and if the        viewer is not a registered viewer, then use one of the following        techniques as a default;    -   Simple deep tag—a specified amount of the movie is captured        before or after button 741 is clicked, with the amount being        determined by the owner of the website, the user, or the owner        of the movie;    -   Duration deep tag—the portion of the movie that plays while        button 741 is being clicked on is captured; thus, beginning to        click the fast deep tag button serves as a first indication and        ending clicking of the fast deep tag button serves as a second        indication;    -   Bracket deep tag—after button 741 is clicked, a first amount of        the movie preceding the click and a second amount of the movie        subsequent to the click are captured, with the first and second        amounts being determined by the owner of the website, the user,        or the owner of the movie;    -   Frame deep tag—after button 741 is clicked, a single frame is        captured, such as the frame being displayed when the button is        clicked, or the frame indicated by positioner 730;    -   Tail deep tag—after button 741 is clicked, a deep tag is created        encompassing the portion of the movie from the click until the        end of the movie;    -   Two click deep tag—the first click of button 741 indicates the        start of a new deep tag, the movie continues to play in screen        2300, and the second click of button 741 indicates the end of        the deep tag.

FIG. 42 is a diagram of pop-up window 2350 for creating a deep tag.Window 2350 appears after a fast deep tag has been created, as describedabove. If the user does not wish to send the just-created deep tag toanyone, the user clicks on the cancel button. Otherwise, the userprovides data and clicks on the send button. The provided data includesthe sender's address, the recipient's address, the desired accesspermission, and whether instant deep tag sending is enabled.

Access permission refers to whether and how the recipient can alter thedeep tag. At one level, no alterations are permitted. At the standardlevel, the recipient can add comments but cannot otherwise alter thedeep tag. At another level, the recipient can create their own versionof the deep tag, indicating more or less of the original movie andpossibly indicating another movie, and inheriting the text associatedwith the first deep tag.

Instant deep tag sending is discussed below with regard to FIG. 43. Ifthis is not enabled, the deep tag can be sent only as a hyperlink in anemail message.

In one embodiment of movie system 10, a deep tag creator is either ananonymous user, a casual user or a formal user. Movie system 10 canconsider deep tags created by some or all of anonymous, casual andformal users when determining the popularity or other statistic of amovie.

An anonymous user is not known to movie system 10. Deep tags created byan anonymous user are generally not stored for subsequent retrieval bythe anonymous user, that is, they exist only while the anonymous user isinteracting with the movie associated with the deep tag. Deep tagscreated by anonymous users have analytic value to movie system 10. Ananonymous user can comment on and rate only their own deep tags. Moviesystem 10 tracks deep tags from anonymous users via a suitable temporarymechanism, such as a cookie.

A casual user is a user for which movie system 10 knows only his or heraddress, such as email address or instant message ID or telephonenumber. Typically, an anonymous user converts to a casual user byentering an address in the sender portion of pop-up window 2350. Acasual user can comment on and rate only their own deep tags. Deep tagscreated by a casual user can be retrieved by the casual user, generallybecause a so-called cookie is stored on the casual user's access device,or because the casual user provides their address to movie system 10.

A formal user is a user that has registered with movie system 10 via asign-up process involving creating a profile with a registered i.d. anddemographic information relating to the user, and perhaps validatingother of their information such as that their email address is actuallycontrolled by them. Deep tags created by a formal user are stored inknown content owner deep tag database 32 or unknown content owner deeptag database 34. A formal user can comment on and rate movies and deeptags created by other formal users, can upload and deep tag movies, canmake deep tags and movies public, and can search and browse their deeptags and public deep tags created by other formal users. A formal usercan be a content owner. Movie system 10 may periodically invite casualusers to convert to formal users. The demographic information associatedwith formal users makes their activities more valuable to movie system10 than the activities of casual or anonymous users. In variousembodiments of movie system 10, the capabilities and privileges of aformal user may differ.

FIG. 43 is a diagram of a recipient's screen showing reception of aninstant deep tag, that is, an icon that appears on a display and whichcan be clicked on to generate a pop-up window to play the movie segmentindicated by the deep tag. The instant deep tag typically has theaddress of the sender, and sometimes includes descriptive text, if any,provided by the sender either when causing the deep tag to be instantlysent, or associated with the deep tag. A sample format for a pop-upplayer is shown in FIG. 28. In some embodiments, the pop-up player hasthe format shown in FIG. 36. At the conclusion of play, the pop-upplayer may automatically display the home page of the website thatprovides the movie associated with the deep tag. Typically, before theinstant deep tag icon appears on the recipient's display, the recipientis asked whether he or she wishes to accept the deep tag. The recipientmay specify in his or her profile that instant deep tags are permittedfrom certain senders, i.e., a whitelist, and may specify that instantdeep tags are prohibited from certain senders, i.e., a blacklist; thesespecifications operate to automatically accept or refuse an instant deeptag, avoiding the acceptance question.

FIG. 44 is a flowchart showing details of creating and modifying deeptags.

Let it be assumed that movie system 10 has received a request to createa deep tag, such as when a user clicks on deep tag button 740 or fastdeep tag button 741.

At step 2400, movie system 10 determines whether the content owner ofthe movie is known, such as by checking known content ownermovie-id-print database 31 or by recognizing that the movie is stored inmovie storage 20. If the content owner is known, then at step 2405,movie system 10 follows the content owner's rules for deep tag creationrelative to the movie. For example, the content owner may permit deeptags only to be made from a specified version of the movie, not fromother copies thereof. As another example, the content owner may requirethat at the conclusion of play of a deep tagged segment, the playerloads a particular webpage. As a further example, the content owner mayrequire that the deep tag include one or more advertisements included inthe content. As a yet further example, the content owner may require ashare of revenue realized from viewing of the content or any deep taggedportion of the content. Other rules are contemplated such as controllingthe ways in which deep tags can be combined, where some deep tags mightonly be allowed for combining with deep tags from the same contentowner. At step 2410, the deep tag is added to known content owner deeptag database 32, and deep tag creation processing is complete.

However, if the content owner is unknown, then at step 2415, moviesystem 10 notifies the would-be creator that the content owner isunknown and so movie system 10 will be collecting data on who views theto-be-created deep tag, and when, and will be notifying recipients ofthe deep tag that if they click it, data about them will be collected.The notification is provided by a pop-up window, playing an audio file,or other suitable means. In some embodiments, the would-be creatornotifies movie system 10 of who the likely content owner is, such as byselecting from a menu of registered content owners and/or by providingcontact information and/or a name of the content owner; this serves as arequest for permission to use the content, which is generally forwardedby movie system 10 to the likely content owner. At step 2420, it isdetermined whether the would-be creator consents to the data collection.If not, processing is complete. If consent is provided, then at step2425, movie system 10 creates the deep tag and adds it to unknowncontent owner deep tag database 34. Then, at step 2430, movie system 10checks whether the movie has a movie-id-print in unknown content ownermovie-id-print database 33. If so, this means that registered contentowners have already been notified, and so processing is complete. Ifnot, at step 2433, movie system 10 places the movie-id-print to unknowncontent owner database 33, and at step 2435, movie system 10 sends analert to all registered content owners, or to a subset of all registeredcontent owners, and to the owner of the site where the deep tagged movieresides, if the site owner is known. The subset may be determined by thetext associated with the movie, by the similarity of the movie to movieswith known content owners, or by another suitable method.

Registered content owner 141 is a party who has previously registeredwith movie system 10 to receive alerts when a movie with an unknowncontent owner is deep tagged. At step 2480, registered content owner 141receives an alert from movie system 10. At step 2485, owner 141determines whether they own the content, and if so, establishesownership by a suitable procedure. In some cases, content ownership isestablished in response to other than alerts from movie system 10; thatis, a content owner may otherwise become aware that its content is beingused through movie system 10. At step 2490, owner 141 specifies rulesfor deep tagging the movie, and sends the rules to movie system 10.

At step 2450, movie system 10 receives the newly created content owner'srules for deep tagging. If not already done during step 2485, moviesystem 10 transfers the movie-id-print from database 33 to database 31.At step 2460, movie system 10 ensures that deep tags associated with themovie comply with the rules, and if they do not, either (a) the deeptags are suitably adjusted, or (b) the deep tag is rendered inactive,and the deep tag creator is invited to adjust the deep tag to make itactive again. At step 2465, the deep tags associated with the movie aremoved from database 34 to database 32. At step 2470, movie system 10notifies the deep tag creator and all parties who have viewed the deeptag that the content owner is known, and any unauthorized copies shouldbe destroyed or brought into compliance with the content owner's rules.

Dynamically updated viewer displays will now be discussed. First,dynamically updated deep tags will be discussed. Then, dynamicallyupdated movie selections will be discussed.

Dynamically updated deep tags are relevant when a viewer is watching amovie in a manner in which the viewer can see the deep tags associatedwith the movie, such as:

-   -   by viewing a thumbnail associated with the deep tag, shown for        example as deep tag thumbnail 738A in FIG. 35;    -   by viewing a popularity graph of the deep tags for a movie,        shown for example as a color graph in graph area 1394 in FIG.        35, or density function 960 in FIG. 16 (that would be displayed        in graph area 1394), or cloud graph 980 in FIG. 16 (that would        be displayed in graph area 1394);    -   by viewing a composited deep tag thumbnail, shown for example as        deep tag thumbnail 2335 in FIG. 41; or    -   by using a cursor positioning device such as a mouse to view        information associated with deep tags indicated on a texture        strip, such as deep tag imprint 2320 in FIG. 41.

Movie system 10 receives, in real time, information about how a vieweris interacting with a movie, such as clicking the play button, the pausebutton and so on. Accordingly, movie system 10 knows all current viewersof a movie.

When one viewer of a movie acts on a deep tag, such as creating, editingor deleting the deep tag, then the deep tag displays, if any, for allother viewers of the movie are updated to reflect the deep tag act. Thisis dynamically updated deep tag display.

It will be recalled that if the viewer is an anonymous user, the deeptags created by that viewer persist only for the duration of theviewer's session. Specifically, when movie system 10 determines that theanonymous user is no longer watching the movie, movie system 10dynamically updates the displays of all other viewers of the movie todelete any deep tags created by the anonymous user.

In some embodiments, formal users and/or the owner of the movie and/orthe owner of the website that provides the movie can control whetherdynamically updated deep tag displays are supported. Specifically, someformal users may prefer to turn off this feature.

Dynamically updated movie selections enable movie system 10 to operatein a sort of narrowcast format. Different versions of dynamicallyupdated movie selections are discussed. It will be appreciated thatother versions of dynamically updated movie selections can beconfigured.

A first version of dynamically updated movie selections enables a formaluser of movie system 10 to specify characteristics of movies that shouldbe presented to the user, and then the user can just view in a passivemode, without selecting movies. Example characteristics are:

-   -   the most popular movies that I haven't seen yet, in all of movie        system 10    -   the most popular movies that I haven't seen in the last week, in        all of movie system 10    -   the most popular movies that I haven't seen yet, among female        viewers aged 20-25 in New York and London    -   the most popular movies that I haven't seen yet with the keyword        “Paris”

In a variation of this version of dynamically updated movie selections,instead of the whole movie being displayed, only its deep tags withcertain characteristics are displayed, such as its three most populardeep tags, deep tags created by users with certain demographiccharacteristics, or deep tags created by specifically named users.

A viewing trail is a particular sequence of movie viewing for aparticular user. The viewing trail includes not only which movies wereviewed, but also the actions taken for the movie, such as play, pause,fast forward, rewind and so on.

In another version of dynamically updated movie selections, formal userscan designate their viewing trails as public or private. Movie system 10assumes a default of “private” for viewing trails. A formal user canspecify the level of public access to their viewing trails, e.g., opento all, open to anyone who pays, open to designated users (whitelist),open to all except designated users (blacklist), open to users withcertain demographic characteristics, open to all except users withcertain demographic characteristics, and so on.

If a formal user has public viewing trails, then another formal user caninstruct movie system 10 to show the other formal user the movies thatwere viewed by the first formal user, such as in the first formal user'smost recent movie viewing, on a specific day or meeting othercharacteristics. This is useful when the first formal user is acelebrity, or carefully selects their viewing trails. It will beappreciated that a public viewing trail is a simple version of a mash-upmovie. However, the creator of the mash-up usually recognizes revenuefrom ads inserted into the mash-up. The creator of a viewing trail canrecognize revenue from something akin to a subscription model. In mostcases, creation of a viewing trail is easier to accomplish than creationof a mash-up movie. In a variation, the first formal user can create aviewing trail, and then email it to friends.

In a variation of this version of dynamically updated movie selections,instead of the whole movie being displayed, only its deep tags createdby the first formal user are displayed.

Although an illustrative embodiment of the present invention, andvarious modifications thereof, have been described in detail herein withreference to the accompanying drawings, it is to be understood that theinvention is not limited to this precise embodiment and the describedmodifications, and that various changes and further modifications may beeffected therein by one skilled in the art without departing from thescope or spirit of the invention as defined in the appended claims.

What is claimed is:
 1. A method comprising: storing, by a movie system,a video on a non-transitory computer-readable medium, the movie systemembodied and operating a server machine; responsive to input receivedvia a graphic user interface, generating, by the movie system, a deeptag, the graphic user interface providing a texture strip comprisingframe representations of the video, a deep tag marker overlaying thetexture strip and configured for marking segments of the video over thetexture strip, and a deep tag actuator for defining deep tag informationwhich includes a start time and an end time for a segment of the video;storing, by the movie system, the deep tag at a network address foraccessing the segment of the video, the network address containing atleast one of a deep tag identifier identifying the deep tag, a videoidentifier identifying the video, the start time, or the end time forthe segment of the video; providing, by the movie system, a playerapplication to a user device associated with a viewer; and sending, bythe movie system, the deep tag over a network to the user deviceassociated with the viewer, wherein the player application enables theviewer to activate the deep tag to cause the player application to playthe segment of the video associated with the deep tag on the userdevice.
 2. The method according to claim 1, wherein the playerapplication is configured to display thumbnails of deep tags.
 3. Themethod according to claim 1, wherein the deep tag is sent over thenetwork to the user device as a hyperlink.
 4. The method according toclaim 1, further comprising: collecting, by the movie system, statisticsrelating to the viewer viewing the video, deep tag creation, and deeptag usage.
 5. The method according to claim 4, further comprising:generating, by the movie system, a report relating to the statisticscollected by the movie system.
 6. The method according to claim 4,wherein the statistics collected by the movie system include informationrelating to at least one of: a number of deep tags created for a portionof the video, an access frequency for a portion of the video, a replayfrequency for a portion of the video, a pause frequency for a portion ofthe video, and a community score for a portion of the video.
 7. Themethod according to claim 1, wherein the generating the deep tagcomprises applying a deep tagging rule to the input received via thegraphic user interface.
 8. A movie system, comprising: a processor; anon-transitory computer-readable medium; and stored instructionstranslatable by the processor for: storing a video on the non-transitorycomputer-readable medium; responsive to input received via a graphicuser interface, generating a deep tag, the graphic user interfaceproviding a texture strip comprising frame representations of the video,a deep tag marker overlaying the texture strip and configured formarking segments of the video over the texture strip, and a deep tagactuator for defining deep tag information which includes a start timeand an end time for a segment of the video; storing the deep tag at anetwork address for accessing the segment of the video, the networkaddress containing at least one of a deep tag identifier identifying thedeep tag, a video identifier identifying the video, the start time, orthe end time for the segment of the video; providing a playerapplication to a user device associated with a viewer; and sending thedeep tag over a network to the user device associated with the viewer,wherein the player application enables the viewer to activate the deeptag to cause the player application to play the segment of the videoassociated with the deep tag on the user device.
 9. The movie system ofclaim 8, wherein the player application is configured to displaythumbnails of deep tags.
 10. The movie system of claim 8, wherein thedeep tag is sent over the network to the user device as a hyperlink. 11.The movie system of claim 8, wherein the stored instructions are furthertranslatable by the processor for: collecting statistics relating to theviewer viewing the video, deep tag creation, and deep tag usage.
 12. Themovie system of claim 11, wherein the stored instructions are furthertranslatable by the processor for: generating a report relating to thestatistics collected by the movie system.
 13. The movie system of claim11, wherein the statistics collected by the movie system includeinformation relating to at least one of: a number of deep tags createdfor a portion of the video, an access frequency for a portion of thevideo, a replay frequency for a portion of the video, a pause frequencyfor a portion of the video, and a community score for a portion of thevideo.
 14. The movie system of claim 8, wherein the generating the deeptag comprises applying a deep tagging rule to the input received via thegraphic user interface.
 15. A computer program product comprising anon-transitory computer-readable medium storing instructionstranslatable by a processor of a movie system for: storing a video;responsive to input received via a graphic user interface, generating adeep tag, the graphic user interface providing a texture stripcomprising frame representations of the video, a deep tag markeroverlaying the texture strip and configured for marking segments of thevideo over the texture strip, and a deep tag actuator for defining deeptag information which includes a start time and an end time for asegment of the video; storing the deep tag at a network address foraccessing the segment of the video, the network address containing atleast one of a deep tag identifier identifying the deep tag, a videoidentifier identifying the video, the start time, or the end time forthe segment of the video; providing a player application to a userdevice associated with a viewer; and sending the deep tag over a networkto the user device associated with the viewer, wherein the playerapplication enables the viewer to activate the deep tag to cause theplayer application to play the segment of the video associated with thedeep tag on the user device.
 16. The computer program product of claim15, wherein the player application is configured to display thumbnailsof deep tags.
 17. The computer program product of claim 15, wherein thedeep tag is sent over the network to the user device as a hyperlink. 18.The computer program product of claim 15, wherein the instructions arefurther translatable by the processor for: collecting statisticsrelating to the viewer viewing the video, deep tag creation, and deeptag usage.
 19. The computer program product of claim 18, wherein theinstructions are further translatable by the processor for: generating areport relating to the statistics collected by the movie system.
 20. Thecomputer program product of claim 18, wherein the statistics collectedby the movie system include information relating to at least one of: anumber of deep tags created for a portion of the video, an accessfrequency for a portion of the video, a replay frequency for a portionof the video, a pause frequency for a portion of the video, and acommunity score for a portion of the video.